Multimodal LLMs that recognise events reliably still fail to say when they happen. Prompted for timestamps, strong VLMs reach as little as $3.8\%$ R@0.5 on Charades-STA, and $77$ to $80\%$ of their wrong predictions carry low output entropy: the models are confidently wrong, and entropy-based error detection stays below a random classifier. We show that this failure lives in the task interface, not in perception. Holding the weights fixed, replacing timestamp regression with a coarse-to-fine scan of binary questions, whose first-token probabilities are consumed only as a ranking, raises R@0.5 by $28$ to $50$ points across four frozen backbones. The residual failures decompose into two measurable axes: a perception axis that moves with the backbone, and a geometry axis that is analytically predictable from the ratio of the output-window and event widths. FV-Action, the training-free method built on this analysis, reaches $56.8\%$ R@0.5 on Charades-STA, above the same backbone's native grounding pipeline and the strongest training-free result on this benchmark; it surpasses every TVG-trained model evaluated zero-shot on TACoS, and improves over direct prediction on ActivityNet Captions and QVHighlights, with no temporal supervision at any stage.
Question answering over visually structured documents remains difficult when evidence is distributed across prose, tables, figures, captions, visual layout, and document structure. We present DOCLAYOUT-MM-RAG, a layout-aware annotation framework for grounded question answering over documents. The framework links each question-answer instance to supporting layout elements, preserving element-level provenance for annotation, retrieval, citation, generation, and evaluation. We instantiate the framework on annual reports and release an initial curated corpus of 650 accepted grounded question-answer instances across 30 documents. The corpus captures evidential complexity, with 36.6% of instances requiring cross-page support and 40.8% requiring multimodal support. Exploratory analyses compare flat-text, structure-aware, and layout-derived multimodal retrieval representations, and show how element-level provenance enables retrieval-to-generation and oracle-evidence analysis. DOCLAYOUT-MM-RAG provides a concrete basis for studying provenance-preserving retrieval-augmented generation over visually structured documents.
Andrew Brown, Christopher Baker, Karen Rafferty et al.· Proceedings of the 2026 ACM...· 0 citations