How Should Multimodal Information Be Chunked for Complex Question Answering?

Abstract

Effective question answering over heterogeneous documents requires not only powerful reasoning models but also careful decisions about how evidence is structured and retrieved. In many real-world settings, information is inherently multimodal: a researcher assessing a scientific finding must reason jointly over paragraphs, figures, and result tables; a financial analyst combines textual summaries with tabular data; and a clinician interprets diagnostic imaging alongside patient records. Yet the question of how to decompose such heterogeneous documents into meaningful, retrievable units of evidence remains largely unexplored. In text-only settings, the shift from document-level to passage-level retrieval has demonstrated that retrieval unit granularity is a consequential design choice, with finer-grained units consistently improving both retrieval precision and downstream question answering (QA) performance. However, this line of research has matured almost exclusively in unimodal settings. Current approaches to multimodal retrieval, whether based on joint embedding spaces or modality conversion pipelines, largely inherit the same assumption: that retrieval units are determined by document structure rather than by the informational needs of the query. Multimodal large language models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, but operate over fixed context windows and are costly to deploy at scale, with performance known to degrade when relevant evidence is distributed across long contexts. Across all these paradigms, the design of retrieval units, specifically how heterogeneous documents should be decomposed and grouped into focused, indexable fragments, remains underexplored. This doctoral research addresses the modality representation gap by investigating how multimodal documents should be decomposed for complex question answering. Drawing on the notion of information nuggets, which have long served as atomic units of evidence in QA evaluation, we ask whether operationalising this principle at the indexing stage can improve both retrieval effectiveness and downstream answer quality. We introduce the notion of Composite Chunk Items (CCIs), semantically coherent, self-contained evidence units that may span text, tables, images, and charts, and investigate how their construction, selection, and presentation to an LLM reader affect end-to-end QA performance. A central question is what constitutes the right level of granularity for such units: too fine-grained and coherent evidence is fragmented; too coarse and the noise of document-level retrieval is reintroduced. Preliminary experiments provide early evidence that chunk-level retrieval consistently outperforms document-level baselines across a range of evidence token budgets, motivating a systematic investigation into CCI design as an independent research variable. The proposed research is structured around three directions: how CCIs should be constructed to capture key facts with minimal redundancy; how retrieved chunks should be selected and ranked across heterogeneous modalities; and how evidence should be packaged and presented to maximise LLM reasoning quality. Rather than proposing a new model architecture, this work treats retrieval unit design as a principled empirical research focus, seeking to clarify how structural decomposition of multimodal documents influences robustness, interpretability, and overall QA quality.