SmolDocling, a compact 256M-parameter vision-language model (VLM), is fine-tune to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing.
Abstract
Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.
End-to-end OCR systems based on vision-language models have achieved strong performance in complex document OCR, but their efficiency is limited by the large number of visual tokens produced from document images. Many of these tokens correspond to blank margins or visually redundant regions, yet directly applying generic visual token compression methods may remove OCR-critical fine-grained details. In this paper, we propose LayoutLite, a lightweight plug-and-play module for efficient document OCR. Instead of relying on explicit document layout detection, LayoutLite performs implicit layout analysis at the token level between the vision encoder and the language decoder. It aggregates multi-layer visual representations from the vision encoder, and predicts an importance score for each visual token with a lightweight scoring network. Low-information tokens are then removed before entering the language decoder while preserving the original spatial positional information of retained tokens. To train LayoutLite without human annotations, we cast token selection as a reinforcement learning problem and optimize it with a group-relative policy optimization objective driven by OCR output consistency, together with an auxiliary layout supervision signal to stabilize training. Experiments on OmniDocBench demonstrate that LayoutLite can substantially reduce visual token length and inference cost with negligible degradation in recognition quality. We further evaluate LayoutLite on two OCR-specialized VLMs, FireRed-OCR and Logics-Parsing-V2. Under up to 50% token compression, LayoutLite preserves almost the same score on both models while reducing prefill latency, FLOPs, and KV cache memory by over 40%, with only a small additional inference overhead. These results show that token-level implicit layout analysis is an effective and practical approach for accelerating VLM-based OCR systems.
Structurally extracting information from mixed-script documents that interleave standard text with open, compositional symbol systems is challenging for both optical character recognition (OCR) and vision–language models (VLMs). This difficulty is epitomized by Jianzi Pu—the ancient Guqin tablature. Unlike closed-set scripts, Jianzi glyphs are formed via infinite compositional rules and are densely integrated with Hanzi text, demanding a model that can simultaneously perform script discrimination, layout parsing, and structural transcription. We propose JZ-Tab, the first framework dedicated to the automated recognition of Jianzi Pu, which functions as an end-to-end structured visual information extraction system for mixed-script documents. Unlike traditional pipelines, JZ-Tab generates layout-aware markup directly from full-page images, bypassing the need for pre-segmentation. Specifically, to overcome the total absence of large-scale annotated datasets, we develop a novel, scalable synthetic-to-real pipeline that constructs layout-consistent pages from canonicalized glyph inventories. Furthermore, to capture the unique action-oriented semantics of the tablature, we introduce music-structured generation, injecting sequential regularities derived from symbolic music logic into the learning process. Finally, we train a VLM for direct page-to-markup generation. Evaluated zero-shot on authentic historical manuscripts, JZ-Tab improves F1 by +40.10 over the strongest generic VLM baselines, highlighting its potential for large-scale automated digitization of historical Guqin manuscripts and open, compositional symbol systems.
Zehan Li, Fu Zhang, Zhijun Liu et al.· Proceedings of the 32nd ACM...· 0 citations
Key Information Extraction (KIE) converts visually rich documents into structured data, but practical deployment remains challenging: strong performance often relies on costly on-server Large Multimodal Models (LMMs), while compact locally deployable models lack sufficient KIE supervision. We present SAYRE, a scene-aware document synthesis framework for generating scalable KIE training data without hand-crafted template design. Given a few exemplar documents, SAYRE captures category-specific content patterns and layout conventions to synthesize document-schema-annotation triples. It further introduces error-driven generation, which expands real-world failure cases into hard training examples while preserving their structural patterns. Experiments on constrained- and open-category KIE show that SAYRE consistently improves Qwen3-VL backbones and achieves the strongest overall performance among on-device LMMs. Data scaling experiments show an overall upward trend as more synthesized data is introduced, especially for smaller models and open-category extraction. Error analysis further shows that synthesized training reduces field-level errors by improving schema-aware extraction over dense tables, business identifiers, and contract clauses. These results establish scene-aware synthesis as an effective data-centric approach for improving practical multimodal KIE.
Zhipeng Xu, Zulong Chen, Qing Liu et al.· 0 citations
M3Grounder sets a new state-of-the-art in grounded DocVQA, advancing from coarse boxes to hierarchical, fine-grained and contextually grounded mask evidence.
Venkata Kesav Venna, Sai Madhusudan Gunda, Jyothi Swaroopa Jinka et al.· 1 citation
Document parsing aims to transform unstructured documents into structured and machine-readable representations. Recent advances in Vision-Language Models (VLMs) have significantly advanced document parsing. However, existing approaches still face two major challenges. First, decoupled VLM-based methods heavily rely on accurate layout analysis, where geometric distortions in camera-captured documents can introduce cascading errors. Second, although end-to-end VLM-based methods alleviate the dependence on explicit layout detection, they often suffer from redundant generation, hallucinations, and insufficient structural reasoning in high-resolution scenarios. To address these challenges, we propose NaviDC-OCR, a unified framework for document parsing. NaviDC-OCR introduces deformation-aware learning to incorporate geometric perception into VLMs and proposes an adaptive sampling mechanism for complex layout representation. Furthermore, a content-structure decoupled learning strategy is developed to explicitly model formula grammars and table structures, enabling more effective structured representation learning. Extensive experiments demonstrate that NaviDC-OCR achieves state-of-the-art performance across diverse document parsing benchmarks. It obtains overall scores of 96.87, 88.53 and 78.41 on OmniDocBench v1.6, Wild-OmniDocBench, and PureDocBench, respectively, and ranks first in the ICDAR 2026 Sci-ImageMiner Challenge. These results validate the effectiveness and generalization capability of NaviDC-OCR in complex document parsing scenarios.
Peng Cai, Zhaofan Zou, Shifa Liu et al.· 0 citations
To tackle the challenges in fine-grained retrieval stemming from noise, official seal occlusions, small text blocks, and other issues prevalent in archival text images, and to fulfill the requirements of integrating both textual and visual dual features while enhancing retrieval accuracy and efficiency, this study has devised a five-tier architectural model. This model comprises an input layer, a preprocessing layer, a scene graph generation layer, an attention fusion layer, and a retrieval matching layer. The model incorporates a dedicated scene graph generation module tailored for archival data, aiming to enhance element detection. Additionally, it features a three-tier attention fusion module that integrates scene graph, text, and cross-modal features to ensure precise feature alignment. Training is carried out using a multi-task loss function, and an index is created to streamline retrieval and matching processes. Experimental results show that the proposed model achieves a Top-1 accuracy of 83.7% and an average precision of 88.3% on the test set, representing a 25.1% improvement over the Top-1 accuracy of an optical character recognition (OCR) combined with word frequency and inverse document frequency model. The proposed model achieves a Top-1 accuracy of 6.1% higher than the archival retrieval network model for examples with official seal occlusion and a Top-1 accuracy of 76.8% for small text blocks. The response time for a single retrieval is 52.6ms. Research provides technical support for efficient retrieval of large-scale archives in archives, effectively solving the problem of archive retrieval in complex scenarios, and significantly improving the efficiency of archive management and utilization.
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.