DeepResearchGym: A Free, Transparent, and Reproducible Sandbox for Deep Research
TL;DR
The sandbox provides a search API that indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN and achieves comparable latency to popular commercial APIs while ensuring stable document rankings across runs.
Abstract
Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries, and/or answers to hard-to-locate factual questions. However, most existing systems rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges, in addition to high costs. To address these limitations, we introduce DeepResearchGym as a free and open-source search sandbox for reproducible research on deep research systems. The sandbox provides a search API that indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves comparable latency to popular commercial APIs while ensuring stable document rankings across runs. We demonstrate the sandbox's utility through two use cases. For training, we synthesize queries grounded in the indexed corpora and show that search agents trained within the sandbox generalize to commercial search at inference time, enabling cost-effective reinforcement learning. For evaluation, we extend the Researchy Questions benchmark with LLM-as-a-judge metrics to measure alignment with users' information needs, retrieval faithfulness, and report quality. Evaluation results show that system rankings remain consistent when switching from commercial APIs to ours.