LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.
This comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection and highlighting that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.
Paula Cordero Encinar, taylan. cemgil, Arnaud Doucet et al.· arXiv.org· 0 citations
PruneShift, an evaluation framework that separates broad predictive fidelity, fidelity near selector outputs, and the quality of the selected pruning decision, is introduced, showing why predictive fit, decision reliability, and pruning method quality require separate evidence.
Much causal inference research is focused on methods for optimizing dynamic treatment regimes (Murphy, 2003; Robins, 2004; Schulte et al., 2015), which are rules for deciding which treatments should be assigned and when based on evolving history. There is a certain optimism underlying this endeavor that with enough tin...
Simulations against group-sequential and always-valid frequentist baselines show that flat-prior posterior stopping exactly reproduces naive peeking, that a well-calibrated empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect varia...
Confidence interval estimation for binomial proportions remains challenging in finite samples due to the discreteness of the data, which induces oscillatory coverage behavior. Classical intervals are often conservative (e.g., Clopper–Pearson) or may exhibit undercoverage (e.g., Wald), and exact nominal coverage cannot...
F. Konietschke, E. Brunner· Statistical Papers· 0 citations
The results indicate that the selection kernel affects operational behavior: concentrated kernels tend to produce more stable accuracy and reduce interaction variability, whereas flatter kernels increase exploration and may prolong sessions.
Aniko Apro, T. Tajti· Annales Mathematicae et Info...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
