Skip to content
Preprint

FormBench: A Benchmark for Fine-Grained Retrieval in Formulation Patents

Oct 2026 · 0 citations · 36 references
Computer Science

TL;DR

FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents, is introduced and 10 model configurations spanning commercial API encoders, open-source encoders, and their task-adaptive fine-tuned (TAPT) variants are evaluated.

Abstract

We introduce FormBench, a platform for testing the ability of models to retrieve precise chemical information from technical documents. FormBench contains 55,352 queries and two retrieval corpus variants each with ~1M passages: FormBench-Random (C0), with randomly sampled distractors, and FormBench-Structured (C1), with structurally proximate distractors. Each query is paired with an anchor passage, a hard negative, and a soft negative - derived from a materials ontology through entity overlap scoring, providing a deeper graded signal on a model's discriminatory power. We evaluate 10 model configurations spanning commercial API encoders (OpenAI, Voyage), open-source encoders (Nomic, GTE-large, Qwen3-Embed), and their task-adaptive fine-tuned (TAPT) variants. We find: (1) commercial APIs score very poorly across all configurations (OpenAI-3-large: 0.009 on C1, 0.213 on C0); (2) TAPT on FormBench triplets lifts Qwen3-Embed to 0.4085 on C1, 0.4835 on C0; (3) BM25 remains competitive (0.375 on C1, 0.467 on C0), exceeded only by TAPT-Qwen3; (4) corpus size has a significant impact on retrieval behavior, as demonstrated through a corpus-size ablation; (5) the API failure traces to within-domain geometric collapse; (6) APIs outperform even the TAPT variants on classification and clustering tasks. FormBench is released in BEIR-compatible format at huggingface.co/datasets/vinven7/FormBench under CC-BY-4.0.

View source

Similar papers

Preprint Oct 2026

xPRT: Cross-Passage Retrieval Pretraining

Dense retrievers are a foundational component underlying modern RAG and search systems. Training these models typically requires a curated collection of annotated query-document pairs, which are prohibitively expensive to obtain outside a handful of well-resourced domains. This has motivated a line of work that leverag...

Danush Khanna, Himanshu Singhal, Pranjal A. Chitale et al. · 0 citations
#small language model Open access Oct 2026

RETRIEVAL BEATS FINE-TUNING: ADAPTING A KAZAKH SMALL LANGUAGE MODEL FOR HISTORICAL QUESTION ANSWERING

Large language models (LLMs) have shown a strong ability to solve a wide range of language tasks. However, they became less practical for subject-specific applications, especially when dealing with low-resource languages. This paper shows an example of adaptation of small language models (SLMs) to solve subject-specifi...

G. Askar, A. Mutaikhan, M. Aitkozhayeva et al. · 0 citations
Preprint Oct 2026

CA-HNM: Constraint-Aware Hard Negative Mining for Dense Retrieval with Structured Domain Models

CA-HNM is introduced, a constraint-aware hard-negative mining framework that combines retrieval-based candidate generation with deterministic incompatibility rules derived from a structured domain model and the results indicate that the contribution of constraint-aware selection depends on the training objective and th...

N. Lê, Marie-Hélène Abel, Bertrand Laforge · 0 citations
Preprint Oct 2026

Retriever-Free Retrieval: Navigating Documents with Decision Models

While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabiliti...

Chao Wan, Paul J. Jünger, Bei-Xi Du et al. · 1 citation · ⚡1
Conference Open access 2026

Comparison and Dynamic Fusion of RAG One-Stage Retrieval Methods for Heterogeneous Queries

Retrieval-Augmented Generation (RAG) strengthens the factuality of large language model outputs by grounding the generation process in retrieved external knowledge. Given that the efficacy of one-stage retrieval strictly defines the evidence available for generation, it is imperative to analyze the applicable boundarie...

Yi-Di Chen, Tian-Lai Feng · 0 citations
Preprint Oct 2026

RE-SciVQA: Improving Scientific VQA With Context-Relevant Retrieval and Lightweight MLLM Adaptation

This work proposes RE- SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training, and examines closed-ended infinite visual questions, a category in which chart-understanding models often struggle.

Sheikh Tahmim Mehdi, Muminul Islam Jami · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.