Skip to content

Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model

Sep 2026 · 11 citations · ⚡ 2 influential · 56 references
Computer Science

TL;DR

An open implementation of JevLite of this readout, JevLite, on scam-call screening, with a fine-tuned ModernBERT encoder and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.

Abstract

Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing ope...

Jonathan Geuter, Andrew Cheng, David Alvarez-Melis · 11 citations
#artificial intelligence Preprint Sep 2026

Evaluating and Benchmarking the System One Model Jev

Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder.

Tobias Deußer, Rajkumar Ramamurthy, L. Sparrenberg et al. · 33 citations · ⚡7
Preprint Oct 2026

To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges?

Decision models such as TypeSafe's Jev answer typed questions about a text (yes/no, multiple choice, or an ordinal score) with calibrated probabilities in a single forward pass and without generating text, which makes them an inexpensive candidate for LLM-as-a-judge workloads. We evaluate 14 System-1 decision models, s...

S. Subramanian · 0 citations
Preprint Sep 2026

JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges

Security evaluation of tool-using agents requires judging actions in context, yet generative judges add latency, explanation overhead, and output-validation failures. We study whether JEV, a typed decision model, offers a useful alternative for retrospective trace classification. We evaluate JEV and four generative jud...

Zhi-Qiang Wang, Yi-Chao Gao · 5 citations · ⚡2
Preprint Oct 2026

System X: An LLM Decision Model That Replies from Its Own KV Cache

System X, a large language model (LLM) decision model that adds a text-reply option to the LLM-as-Jev readout, is introduced, and reply comparisons also differ in prompts and generation limits.

Syed Usama Khalid Bukhari, Syed Safwan Khalid, M. Aamir · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.