Preprint
Sep 2026
Counterfactual fairness audits of clinical language-model agents report a flip rate: how often an action changes when only the patient's demographic descriptor changes. Part of that rate is not demographic. A stochastic agent also changes its own action when nothing changes, and a flip rate cannot be interpreted withou...
Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar et al.
· 0 citations
Save
{ copied = true; setTimeout(() => copied = false, 1500) })"
class="icon-btn" aria-label="Copy link">
{ copied = 'apa'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy APA
Copied ✓
{ copied = 'mla'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy MLA
Copied ✓
{ copied = 'bibtex'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy BibTeX
Copied ✓
Whether a judge returns the same verdict when the same request is worded differently remains largely unexamined, so decoding noise is not charged to wording, and the release lets a reader ask the same of any judge not in the roster.
Rohith Reddy Bellibatlu, Edward Raff, Wen-Bin Zhang
· arXiv.org · 11 citations
Save
{ copied = true; setTimeout(() => copied = false, 1500) })"
class="icon-btn" aria-label="Copy link">
{ copied = 'apa'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy APA
Copied ✓
{ copied = 'mla'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy MLA
Copied ✓
{ copied = 'bibtex'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy BibTeX
Copied ✓
It is established that action-level divergence exists and can pass unrecorded by the score, not that any rate generalises.
Rohith Reddy Bellibatlu, Manpreet Singh, Zhou-Tian Han et al.
· 0 citations
Save
{ copied = true; setTimeout(() => copied = false, 1500) })"
class="icon-btn" aria-label="Copy link">
{ copied = 'apa'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy APA
Copied ✓
{ copied = 'mla'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy MLA
Copied ✓
{ copied = 'bibtex'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy BibTeX
Copied ✓
Counterfactual audits are the standard tool for checking whether a clinical agent treats demographically distinct but clinically identical patients differently. They report a flip rate: how often an action changes when only the patient descriptor changes. We show that this quantity is uninterpretable on its own. Re-run...
Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar et al.
· 0 citations
Save
{ copied = true; setTimeout(() => copied = false, 1500) })"
class="icon-btn" aria-label="Copy link">
{ copied = 'apa'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy APA
Copied ✓
{ copied = 'mla'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy MLA
Copied ✓
{ copied = 'bibtex'; setTimeout(() => { copied = null; open = false }, 1000) })"
class="flex w-full items-center justify-between rounded-lg px-3 py-2 text-left text-sm hover:bg-gray-100 dark:hover:bg-ink-800">
Copy BibTeX
Copied ✓