TongGuOCR is proposed, a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents that outperforms representative traditional task-specific OCR models, general-purpose MLLMs, and OCR-oriented MLLMs.
Abstract
Chinese historical documents preserve valuable cultural heritage, but many collections remain accessible only as scanned page images, preventing full-text retrieval, collation, and computational analysis. Optical character recognition (OCR) can bridge this gap, but accurate transcription remains challenging because historical documents often contain complex layouts, rare characters, and nontrivial reading orders. We propose TongGuOCR, a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents. First, a Layout-Aware Preprocessing module constructs and refines locally coherent recognition blocks to preserve local context while reducing interference across regions. Second, a Token-Augmented Recognition module augments the transcription target at two complementary levels: character-level vocabulary expansion gives each rare glyph a direct one-token representation and shortens its decoding path, while line-to-line transition modeling injects discrete spatial displacement tokens that guide the decoder along complex reading paths without requiring precise coordinates. Experiments on two Chinese historical document OCR benchmarks show that TongGuOCR outperforms representative traditional task-specific OCR models, general-purpose MLLMs, and OCR-oriented MLLMs. On the more challenging M5HisDoc benchmark, TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric. An online demo is available at https://jzzh2004.github.io/TongGuOCR.
Results show that token-level implicit layout analysis is an effective and practical approach for accelerating VLM-based OCR systems, and proposes LayoutLite, a lightweight plug-and-play module for efficient document OCR.
SPIRAL (Self-improving Path Integration and Realignment), a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision, requiring no external teachers or additional annotations, generalize to out-of-domain benchmarks, confirming that effective VTC hinges on aligning rendered-image representations back to native-text semantics.
Tianyu Liang, Xiangxi Zheng, Yilin Wang et al.· 0 citations
PaDoc is proposed, a layout-grounded parser that treats the predicted layout as a branching structure over a shared page representation that is the fastest end-to-end parser at five concurrency levels and is the fastest end-to-end parser at five concurrency levels.
Artistic Text Recognition (ATR) remains challenging because word images often combine decorative fonts, curved layouts, object-like characters, clutter, and severe distortions. This paper studies WordArt-V1.5 as a standardized benchmark for this setting and evaluates recent scene and artistic text recognizers under a common protocol. We propose a confidence-aware ensemble that combines SVTRv2, PARSeq, and MAERec after fine-tuning on the official training split. The ensemble selects predictions using the minimum confidence over disagreement positions, emphasizing characters that separate competing hypotheses. For long words, where a single character error can invalidate the whole prediction, we add a targeted refinement stage based on Needleman-Wunsch alignment and lexicon-guided correction. On the WordArt-V1.5 Test B split, the proposed system reaches 89.90% Word Recognition Accuracy, improving the best individual fine-tuned model by 1.77 percentage points. The long-word refinement produces a modest global gain, but improves the targeted long-word subset by 2.72 percentage points. Finally, an error analysis of all remaining mistakes shows that 48.8% are associated with labeling issues, visual ambiguity, or illegible samples, highlighting the value of diagnostic reporting for future ATR benchmarks and models. Our source code is available at https://github.com/lucas-azdias/Artistic-Text-Recognition/.
L. A. Dias, Henrique A. Schulz, Rafael Tadeu Machado de Miranda et al.· 0 citations
Scene Text Retrieval (STR) aims to search images containing a given textual query within large-scale image collections. However, existing approaches are fundamentally constrained in two ways: 1) they are evaluated on narrow benchmarks that focus primarily on natural scenes; and 2) they rely on either error-prone multi-stage recognition-then-matching pipelines or localization-assisted matching strategies. To address these limitations, we introduce MuST, the first comprehensive benchmark for multi-scene and bilingual STR tasks, covering a broad range of real-world scenarios with carefully curated Chinese and English textual queries. On top of this benchmark, we propose BPE-Ret, a novel Byte-Pair Encoding (BPE)-level retrieval framework built on a simple yet powerful principle: a word is considered present in an image if and only if all of its constituent subwords are present. Concretely, BPE-Ret decomposes textual queries into BPE subwords and directly aligns them with dense visual features within a unified embedding space, thereby eliminating the need for explicit text spotting and coarse-grained word-level matching. We further enhance fine-grained alignment through two key innovations: a weighted preference learning scheme that prioritizes challenging cases to sharpen discrimination on confusable word-image pairs, and a subword inclusive-OR matching strategy that enforces constituent subword verification to enable robust retrieval beyond word-level granularity. Extensive experiments show that BPE-Ret establishes new state-of-the-art performance on both existing public STR benchmarks and our newly proposed MuST dataset, demonstrating the effectiveness and robustness of subword-level retrieval for real-world, multilingual scene text understanding.
Tong-Kun Guan, Yu-Tong Cai, Haocheng Wang et al.· IEEE Transactions on Image P...· 0 citations
KoVRE: Korean Visual Document Retrieval Embedding is introduced, a single-vector retriever for Korean visual documents, alongside a comprehensive training recipe, demonstrating that targeted bilingual supervision and carefully designed training strategies can produce a highly effective Korean VDR model across diverse document domains, without requiring a scaled-up backbone or multi-vector representations.
