Seyed Mosayeb Alam

What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built...