An end-to-end method for extracting financial document data using transformers

Abstract

This study introduces an end-to-end multimodal transformer framework for extracting structured data from complex financial documents. By using a cross-modal encoder, the system can simultaneously process narrative text, hierarchical tables, and embedded visual elements. This enables the system to model complex semantic relationships and document layouts. A dedicated preprocessing pipeline aligns modalities at the feature level, addressing issues of multilingual content, variable template formats, and document noise. Unified multi-task decoding simultaneously predicts entities, relationships, and values, thereby improving overall extraction accuracy and supporting adaptation to irregular document structures. Evaluated on a large, annotated real-world dataset of invoices and reports. The results indicate that, compared to text- and table-centric baselines, the proposed method demonstrates superiority with significant improvements in F1 score and extraction robustness. Ablation analysis also revealed the crucial role of modality alignment and cross-modal attention in achieving consistent results under high variability. Comprehensive error analysis demonstrated the system's resilience to incomplete inputs and its effectiveness in resolving cross-modal ambiguities. This framework provides a practical and scalable solution for enterprise applications to automatically process financial documents.