Skip to content

Agents That Edit Documents: Measuring Agentic PDF Forgery Against a Non-Agentic Control

Sep 2026 · 2 citations · 31 references
Computer Science

TL;DR

AgentForge-Bench measures how reliably an off-the-shelf coding agent, driving one of seven open-weight models with a shell and the stock Python PDF stack, alters one dollar amount, date or address in a real filed financial document from a single sentence of intent, graded by rules rather than by a model.

Abstract

AI agents that carry a multi-step computer task through on their own became ordinary tools in the past year, and the same autonomy is available to anyone whose task is harmful. We ask what that means for a relying party -- an insurer, a lender, an auditor -- whose evidence is a filed PDF. AgentForge-Bench measures how reliably an off-the-shelf coding agent, driving one of seven open-weight models with a shell and the stock Python PDF stack, alters one dollar amount, date or address in a real filed financial document from a single sentence of intent, graded by rules rather than by a model. Across 1,750 cells, 1,419 (81.1%) satisfy the verifier, and 808 (46.2%) also survive every stricter filter: visible, localized, typeface-matched, original value gone document-wide. A deterministic script with no model in it solves 98 of the 125 documents; the agents solve 124, and none the script solves alone. Agents misreport 41% of their wrong edits as done, no model refused, and the cheapest verified forgery costs 2.4 cents. The raw rate overstates the threat by about a factor of two; the strict rate is still large.

View source

Similar papers

Preprint Oct 2026

Soei: Structured Observability of Evidence and Intent for Multi-Agent AI Software Engineering

AI coding agents now do long, multi-step and multi-session software work with little supervision. When a developer comes back to that work, the useful question is not what the model said but what was actually done, by whom, and whether it is finished. We study this in three parts. First, we verify Agenttrail, an open-s...

Mohammed Nayeem Khan, K. S. Sarun Kumar · 0 citations
Preprint Oct 2026

TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?

Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so i...

Ritiz Tambi, S. Shayegan, Joshua Meyer et al. · 0 citations
Preprint Oct 2026

Infinity Machine: a self-improving AI agent for research

We present the Infinity Machine, a self-improving AI agent that we use for research on problems whose answers aprogram or data can check. Each candidate solution is produced by a full tool-using agent that works for up to an hour in a fresh sandbox; proposers see only a public score, and a private split decides what co...

H. Ganesan · 0 citations
#artificial intelligence Review Oct 2026

Not Self-Decidable: LLMs Cannot Draw the Boundary of What an Agent Verifier Can Check

A verifier for an agent faces rules of two kinds: the ones a fixed check can settle and the ones that require a judge. A team that derives its own checks fixes that split up front. Where the requirements come from outside, as in finance, healthcare and law, the agent enforces rules it did not write, so the split falls...

Anthony Rhodes · 0 citations
#artificial intelligence Review Oct 2026

Holding Agents Accountable with Verifiable Agent Insurance

While AI agents are useful for many tasks, they regularly misbehave in destructive ways, such as deleting important data. While prior work has focused on preventing such misbehavior, we consider how to recover after an agent misbehaves. We propose verifiable agent insurance (VAI), a framework for agent providers to com...

Hua Cheng, Taro Tsuchiya, Nicolas Christin et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.