FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents, is introduced and 10 model configurations spanning commercial API encoders, open-source encoders, and their task-adaptive fine-tuned (TAPT) variants are evaluated.
Abstract
We introduce FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents. FormBench contains 55,352 queries and two retrieval corpus variants each with ~1M passages: FormBench-Random (C0), with randomly sampled distractors, and FormBench-Structured (C1), with structurally proximate distractors. Each query is paired with an anchor passage, a hard negative, and a soft negative - derived from a materials ontology through entity overlap scoring, providing a deeper graded signal on a model's discriminatory power. We evaluate 10 model configurations spanning commercial API encoders (OpenAI, Voyage), open-source encoders (Nomic, GTE-large, Qwen3-Embed), and their task-adaptive fine-tuned (TAPT) variants. We find: (1) commercial APIs score very poorly across all configurations (OpenAI-3-large: 0.009 on C1, 0.213 on C0); (2) TAPT on FormBench triplets lifts Qwen3-Embed to 0.4085 on C1, 0.4835 on C0; (3) BM25 remains competitive (0.375 on C1, 0.467 on C0), exceeded only by TAPT-Qwen3; (4) corpus size has a significant impact on retrieval behavior, as demonstrated through a corpus-size ablation; (5) the API failure traces to within-domain geometric collapse; (6) APIs outperform even the TAPT variants on classification and clustering tasks. FormBench is released in BEIR-compatible format at huggingface.co/datasets/vinven7/FormBench under CC-BY-4.0.
Dense retrievers are a foundational component underlying modern RAG and search systems. Training these models typically requires a curated collection of annotated query-document pairs, which are prohibitively expensive to obtain outside a handful of well-resourced domains. This has motivated a line of work that leverag...
Danush Khanna, Himanshu Singhal, Pranjal A. Chitale et al.· 0 citations
Large language models (LLMs) have shown a strong ability to solve a wide range of language tasks. However, they became less practical for subject-specific applications, especially when dealing with low-resource languages. This paper shows an example of adaptation of small language models (SLMs) to solve subject-specifi...
G. Askar, A. Mutaikhan, M. Aitkozhayeva et al.· NEWS of National Academy of...· 0 citations
CA-HNM is introduced, a constraint-aware hard-negative mining framework that combines retrieval-based candidate generation with deterministic incompatibility rules derived from a structured domain model and the results indicate that the contribution of constraint-aware selection depends on the training objective and th...
N. Lê, Marie-Hélène Abel, Bertrand Laforge· 0 citations
While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabiliti...
Chao Wan, Paul J. Jünger, Bei-Xi Du et al.· 1 citation· ⚡1
Retrieval-Augmented Generation (RAG) strengthens the factuality of large language model outputs by grounding the generation process in retrieved external knowledge. Given that the efficacy of one-stage retrieval strictly defines the evidence available for generation, it is imperative to analyze the applicable boundarie...
Yi-Di Chen, Tian-Lai Feng· ITM Web of Conferences· 0 citations
This work proposes RE- SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training, and examines closed-ended infinite visual questions, a category in which chart-understanding models often struggle.
Sheikh Tahmim Mehdi, Muminul Islam Jami· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.