Skip to content

PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations

Aug 2026 · 0 citations · 39 references
Computer Science

TL;DR

This work introduces PII-Tracer, a compact 0.6B-parameter detector trained with conversation-level supervision that attains the highest entity-level coverage of any system the authors evaluate and also performs strongly on standard single-record benchmarks.

Abstract

LLM assistants and agentic systems log long multi-turn conversations. AI providers often scan these conversations for Personally Identifiable Information (PII) and mask the PII before storing or processing conversation data. Yet most PII detectors and benchmarks target self-contained records rather than cross-turn evaluation. To evaluate PII detection across turns in multi-turn conversations, we introduce PII-TRACE (Tracing Recurring PII Across Conversational Exchanges), to our knowledge the first PII benchmark to assess whether detectors identify PII in conversational contexts and cover every mention of a recurring identifier across turns. PII-TRACE contains 13,148 synthetic multi-turn dialogues in 13 languages with character-level spans and identifier clusters. Across eleven baselines, including frontier LLMs, no detector achieves full entity-level coverage without substantial false positives on PII-free conversations, and single-pass reading loses a third of the gold characters on long dialogues. To close this gap, we introduce PII-Tracer, a compact 0.6B-parameter detector trained with conversation-level supervision. PII-Tracer attains the highest entity-level coverage of any system we evaluate and also performs strongly on standard single-record benchmarks.

View source

Similar papers

#natural language process... Conference Open access Sep 2026

CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work. We introduce CypherTurn, the first benchmark for conversational Text-to-Cypher evaluation, comprising 721 sessi...

Yu-Zhe Zhang, Wei-Jie Zhu, Hao-Lin Yang et al. · 0 citations
#natural language process... Preprint Aug 2026

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection, finds end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in...

Freeman Jiang, Ramon Sanabria, Soham Deshmukh et al. · 9 citations · ⚡4
#human-computer interacti... Preprint Sep 2026

Targeted and Traceable Investigation of Multi-Agent LLM Dialogue via Semantic Bundling of Knowledge Graphs

This work presents a targeted and traceable approach to investigating multi-agent LLM dialogue, applied to the VAST Challenge 2026 MC1 dataset, and finds that the KG structure enables users to identify actors worth investigating faster and summarizing only the region surrounding an actor of interest better supports per...

Ze-Yu Hua, Adam Coscia, Alex Endert · 0 citations
#natural language process... Preprint Aug 2026

You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding

The results show that CoRG remains challenging for current agents, even the best agent reaches only 67.0% success rate, leaving one third of references unresolved, and position CoRG as a concrete benchmark for studying how agents search, inspect, and verify information in realistic multi-tool environments.

Karen Fuchs, Uri Katz, Yoav Goldberg · 0 citations
#natural language process... Preprint Sep 2026

Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. Ensuring their reliability is challenging due to multi-turn interactions, the absence of explicit ground truth, and strict safety constraints. Existing evaluation techniq...

Vaishnav Negi, Lev Sorokin, S. T. Arasteh et al. · 0 citations
#artificial intelligence Book Open access Sep 2026

Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue

These findings show that gradient-based signals can support multi-turn jailbreak detection, but reliable deployment requires calibration on realistic benign conversations, short-window scoring, length-aware thresholds, and evaluation across attack types and model architectures.

Omar Sheta, Rinku Deuja, Hadi Masoudi et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.