OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking

Abstract

Forecasting is a composite capability encompassing information gathering, evidence integration, situational assessment, and decision-making. Large language models (LLMs) combine reasoning and tool use, offering new avenues for automated forecasting and decision support. Yet LLM forecasting evaluation faces a structural challenge: live benchmarks support prospective assessment, but limited question reuse and changing information conditions make results difficult to reproduce and compare; retrospective benchmarks offer reproducibility but risk information leakage. We introduce OracleProto, a reproducible framework for evaluating LLM forecasting in a controlled, temporally masked environment. Through model knowledge cutoff constraints and temporal masking, OracleProto constructs a temporal sandbox for real-world web retrieval, audits information boundaries through leakage detection, and reconstructs resolved events as forecasting instances. Discrete answer normalization and hierarchical scoring enable unified evaluation. We evaluate 27 model-configuration pairs on 300 FutureX-Past-derived questions over 24,300 prediction trials, assessing forecasting quality, sampling stability, and cost efficiency under controlled information boundaries. An audit finds only 1.37% residual leakage in retrieval results after temporal masking and leakage filtering. OracleProto retains resolved forecasting tasks as reusable evaluation resources under explicit temporal boundaries, supporting continual accumulation of forecasting datasets and systematic evaluation of forecasting capabilities. Code is available at https://anonymous.4open.science/r/OracleProto-2E3A/.