Skip to content
Log in
#small language model Open access

CytoGate-Bench: an LLM benchmark for cross-panel cell gating in cytometry

Jaesik Kim Byounghan Lee Namhyuk Ahn Matei Ionita Michelle L. McKeague Matthew E. Lee Chang-Uk Jeong Sokratis A. Apostolidis Amy E. Baxter Shwetank A. Greenplate E. Wherry Kyung-Ah Sohn Dokyoon Kim
Aug 2026 · bioRxiv · 0 citations · 5 references
Biology

TL;DR

This work introduces CytoGate-Bench, a benchmark that reformulates this per-step procedure as a zero-shot, panel-agnostic task for large language models, and contributes a public benchmark that tests precisely that ability across 11 human cohorts.

Abstract

In cytometry, the workhorse single-cell technology of clinical immunology, every study defines its own antibody panel and cell-type vocabulary, so a classifier trained on one cannot annotate the next. Immunologists instead annotate by manual gating, splitting one parent population at a time on a two-marker plot, down an expert-defined hierarchy. We introduce CytoGate-Bench, a benchmark that reformulates this per-step procedure as a zero-shot, panel-agnostic task for large language models. It comprises 23,646 expert-annotated instances re-curated from 11 public flow- and mass-cytometry cohorts spanning eight marker panels. Across six open- and closed-weight backbones, the strongest formulation draws one rectangular gate per candidate and falls within the range of trained, panel-specialized baselines. It degrades less under distribution shift. Walking the hierarchy stepwise outperforms predicting every cell type at once. Ablations trace the signal to the data distribution shape and curated marker priors. However, adding vision or a self-verification loop systematically tightens gates. THE BIGGER PICTURE Immunology laboratories worldwide profile blood and tissue with cytometry, an instrument family that measures dozens of protein markers on millions of individual cells. Before any biology can be read out, every cell must be assigned an identity, a step still dominated by manual “gating,” in which an expert draws boundaries on a sequence of two-marker plots, following a documented, hierarchical protocol. Automating this step has remained difficult because every study measures a different marker panel and names a different set of cell types, so conventional machine-learning models must be retrained for each new study. Large language models (LLMs) promise a different route, a single general-purpose model that reads the expert’s protocol and the data and makes each gating decision directly, with no study-specific training. This work contributes a public benchmark that tests precisely that ability across 11 human cohorts. The result is a statement of feasibility rather than superiority. Off-the-shelf models already score in the range of study-specific trained models and tolerate the day-to-day variation that degrades them. That capability matters most for new or small studies, for which no labeled training data exists. Walking the expert’s hierarchy one decision at a time also outperforms asking the model to name every cell type in a single pass, evidence that the structure of expert practice matters more than the scale of the question. These results come from a deliberately minimal setup, untuned models drawing simple rectangular gates, so we read them as a floor rather than a ceiling. Cytometry-aware training, richer gate geometries, and better-calibrated visual feedback are open avenues, and the benchmark gives that progress a fixed yardstick. Sustained progress would give laboratories analysts that keep pace with evolving marker panels without retraining, while leaving a decision trail an immunologist can audit.

DOI Semantic Scholar
Read PDF

Similar papers

#small language model Open access Aug 2026

PARA: Perception, Action, Reasoning, Adaptation. Four Faculties an Institution Can Revoke

The fourth faculty is Adaptation. Any source rendering it as Reflection is in error, including sources by this author, and the distinction is not cosmetic: reflection is a private act with no external consequence, while adaptation writes to institutional memory, which is why it needs a guardrail and why misnaming it removes the reason for one. No trademark is claimed on PARA or on any of the four faculty names. The construct is offered for use, teaching, assessment, extension and criticism by anyone, with attribution, under CC BY 4.0. An operational agent that watches a system and acts on it is usually described as a perceive-and-act loop, and the description omits the two things an institution needs. It omits the reasoning that justifies an action, which is the only part that can be argued with once the action turns out to have been wrong. And it omits the adaptation that closes the loop, which is where the agent's experience becomes something the institution keeps. PARA names four faculties, each carrying a distinct authority type. Perception has read-only access to system signals and emits structured observations, distinguishing what was measured from what was inferred. Reasoning has read access to observations and runbooks, emits a plan and its justification, and writes nothing at all, which is what makes it safe to give it the widest read access of the four. Action holds the sole authority to change production, through enumerated policy-authorized operations only. Adaptation has write access to institutional knowledge and no write access to production. Two faculties write and two do not, and the two that write are the two that carry guardrails. The substantive requirement is that Adaptation is bounded by the same guardrails as Action, which reads as excessive until the failure it prevents is named. An agent that could both act and rewrite the record of its action could launder its own mistakes into institutional memory, and the institution would then improve its future decisions from a corrected account. Nothing about that is detectable downstream, because the record is the only thing downstream has and there is no second copy to compare against. The failure does not require a deceptive agent: one adapting honestly from a mistaken belief about its own action produces the same result, which makes the guardrail a defence against a normal agent rather than a malicious one. The second requirement is the registry entry that turns a faculty from a description into a contract, carrying the faculty, its allowed actions, its forbidden actions, its governing guardrail and its success metrics. Forbidden actions are named although they are formally the complement of the allowed set, because a reviewer cannot otherwise tell a capability deliberately withheld from one nobody thought of. Success metrics sit in the same entry because the metric is what the agent's optimizer pushes against the guardrail. An agent must not exercise a faculty its entry does not record, and an agent that quietly acquires one usually does so incrementally and with good intent: a reasoning faculty given a small write to make itself useful is an action faculty with no guardrail. The acronym and the loop are in different orders, which the specification states explicitly because the mismatch is a reliable source of confusion. The acronym reads P-A-R-A; the loop runs perception, reasoning, action, adaptation, and reasoning precedes action so that a justification is not constructed afterwards. This is the depth treatment of pattern OP-5 of A Pattern Language for Production LLM Platforms, which is the canonical statement and governs where the two disagree. Documented uses of the full four-part model are emerging rather than established, no implementation unconnected to the author has been evaluated, and the laundering failure is argued rather than observed, which the specification records as a weakness of the argument and not only of the phenomenon. It is a specification, not a certification scheme.

Nabeel A. Khan · 2 citations
#small language model Open access Aug 2026

LifeSciBench: Evaluating Language Models on Realistic, Expert-Level Tasks in the Life Sciences

LifeSciBench is introduced, a benchmark of 750 expert-authored tasks designed to evaluate whether language models can handle realistic life science research work, with each constituent task paired with a human expert-written rubric.

Amelia Liu, Andrew Ho, Anne Marie Droste et al. · 2 citations
#artificial intelligence Preprint Aug 2026

TestifAI: Tomography-Based Testing for Deep Learning Systems

TestifAI, a deep learning testing framework for efficient and accurate estimation of robustness against combinations of perturbations, is proposed and partial model tomography is introduced, a novel approach to reconstructing model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.

Arooj Arif, T. Hartung, E. Botoeva et al. · 1 citation
#small language model Preprint Aug 2026

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

This work proposes a composite metric that combines two orthogonal criteria: information retention and throughput gains and finds that it allocates more resources to the most expressive layers compared to evolutionary search, specialized accelerators, or Shapley-value-based approaches that require expensive approximate inference.

A. Safronov · 1 citation · ⚡1
#small language model Preprint Aug 2026

HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software

HEPToolBench is introduced, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension, and moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models.

Unknown authors · 1 citation

Related blog posts

Microsoft Research Blog Aug 31, 2026

GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models

What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.