Skip to content

FinRegQA-EU: Corruption-Based Preference Data for Grounded EU Financial Regulatory Question Answering

Sep 2026 · arXiv.org · Vol abs/2609.36856 · 0 citations · 29 references
Computer Science

TL;DR

This work proposes an end-to-end pipeline for evaluating and improving LLMs on European financial regulatory question answering and detects a blind spot that matters wherever answers must be verifiable.

Abstract

Large Language Models (LLMs) struggle with region-specific factual knowledge, particularly in financial regulation. While benchmarks such as CFinBench, provide broad coverage of financial knowledge in other regions, no comparable resource exists for European financial regulation. We close this gap by proposing an end-to-end pipeline for evaluating and improving LLMs on European financial regulatory question answering. Our dataset is grounded in the official Q&A corpora of the European Banking Authority (EBA) and the European Securities and Markets Authority (ESMA). We evaluate candidate answers with a pointwise LLM-as-a-Judge protocol using three judges from distinct model families, retain only unanimously judged pairs, and sharpen the rejected side through a taxonomy of regulatory failure modes : law swaps, article swaps, hallucinated citations, and hallucinated text. By fine-tuning on the resulting preference pairs exposes a divergence at the core of our findings: supervised fine-tuning attains the highest judge score of any method yet got the lowest rule-based Citation F1. SFT is producing longer answers with three times as many citations, most of them unsupported. DPO and GRPO offer the better trade-off with more concise answers and the highest citation F1 among fine-tuned models. Standard LLM-as-a-Judge evaluation rewards citation density as evidence of grounding and cannot detect when those citations are fabricated. Thus, we detect a blind spot that matters wherever answers must be verifiable. We release the benchmark and evaluation stack at https://github.com/auliakharis/FinRegQA-EU.

View source

Similar papers

Book Open access Nov 2026

FinMFT: Multi-Source Fine-Tuning of Language Models for Financial News

These findings establish multi-source fine-tuning as a practical framework for coordinating structured, semantic, and market supervision in financial language models whose outputs are empirically evaluated against subsequent market outcomes.

Si-Lu Li, Wenhui Sophia Lu, J. Mulvey et al. · 0 citations
Open access Oct 2026

Do LLM Judges Agree with Humans? Evaluating Financial Commentaries from Material Facts

The rapid adoption of large language models (LLMs) in finance has enabled automated generation of in-domain texts such as earnings summaries, market analyses, and commentary on regulated disclosures. Generating accurate and accessible financial commentary from material facts poses challenges related to domain-specific...

Gabriel Assis, Daniela Vianna, Livy Real et al. · 0 citations
Review Oct 2026

FinLLM-KG: Knowledge Graph-Enhanced LLMs for Financial Sentiment Analysis in Algorithmic Trading

Recent advances in Generative AI have substantially improved financial sentiment analysis through post-trained financial large language models (LLMs). However, existing approaches remain confined to predicting the sentiment of each article in isolation and operate as black boxes, which limits contextual understanding a...

Giorgos Iacovides, Ole Hannibal, Wu-Yang Zhou et al. · 0 citations
Preprint Nov 2026

IFRS Compliance Verification: An Open Multi-LLM Benchmark and Content-Conditioned Ensemble Classifier

Verifying corporate financial disclosures against the International Financial Reporting Standards (IFRS) is a core auditing task in which experts cross-reference disclosures with hundreds of fine-grained regulatory paragraphs: a laborious, costly, and error-prone process that motivates growing interest in large languag...

Aghoghomena Akasukpe, Li Yang, Xin-Yao Zhou et al. · 0 citations
Book Open access Nov 2026

FinCN-QA: Auditing Issuer-Cue Confounding in Open-Corpus Financial RAG

Open-corpus filing QA retrieval evaluation usually reports a single Recall@k, yet issuer-grounded questions mix two stages: identifying the issuer and locating the financial evidence. Existing evaluations often either bypass issuer identification by pre-scoping the corpus to one firm or expose explicit issuer cues in t...

Zhe-Yi Xu, Yan-Jun Yang, Xiang Chen · 0 citations
Book Open access Nov 2026

FinSumm: Towards Analyst-Level Financial Chart Summarization with Human-Aligned Vision-Language Model

Financial chart summarization requires numerical grounding and domain-specific reasoning, yet existing data and general-purpose VLMs provide limited support for this task. We introduce FinSumm-8B, an analyst-oriented VLM trained by supervised fine-tuning and fact-calibrated preference optimization (FCPO), which uses so...

Wei-Mei Wang, Tan Yue, Dong-Yan Zhao · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.