Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers'early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho<= 0.53, p>= 0.11; Bonferroni-corrected Mann-Whitney p>= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers'late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p<0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p<0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.
Active RAG systems decide when to retrieve external knowledge during generation, making them a budget-sensitive case of agentic RAG and self-adaptive retrieval. Yet evaluations often leave the operating point underspecified: two systems may both claim a 50% evidence-usage budget while realizing different held-out usage rates, so higher accuracy can reflect a looser budget rather than a better retrieval policy. We study budget-aware evaluation for Active RAG by recasting active retrieval as utility estimation, where retrieval is valuable only through its marginal correctness change over a no-retrieval answer. This view separates three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side computation changes deployment cost. We operationalize these questions with exact top-k utility frontiers, deployable threshold frontiers, conservative budget frontiers, harm audits, and cost decompositions. Across knowledge-intensive multi-hop QA datasets and open instruction models, retrieval harm is non-negligible, router rankings change across datasets and budgets, nominal thresholds can miss target usage, and simple uncertainty or retrieval-score baselines often rival learned utility routers. Budget-aware Active RAG evaluations should therefore report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.
Pinyan Qian, Su Wang, Chong Peng et al.· arXiv.org· 1 citation
Multi-agent applications delegate work across independently operated deployers. After an incident, a verifier must answer two questions: which deployer released the reported bytes, and whether each cross-deployer edge was authorized. Credentials establish who may act, but need not bind them to later output bytes or prove both deployers authorized a dynamically created edge. We present a two-layer attestation design for dynamic delegation without a shared authority, public log, or precommitted workflow. A trusted deployer runtime signs a hash of each released output; this records released bytes but does not prevent prompt injection. Ancestry evidence records edge authorization. Under a unified threat model, we compare a signed linked list, a Merkle-chain variant, and a co-signed DAG. The primitives are standard; the contribution is deployer-side binding and the evidence needed for the two questions. After child-key compromise, the single-signer designs permit an unauthorized parent binding, whereas the co-signed DAG rejects it because the parent must authorize the edge. Fixed adversary matrices and regression tests validate the composed verifier. On an Apple M1 Pro, ancestry-only checks take 24.3-499.2us per hop. In a live local multi-service workflow, a parent discovers the child's A2A Agent Card; the child calls an MCP tool and releases local-LLM output: all 30 signed-DAG tasks passed complete verification, while a controlled child-key-only claim was rejected; its mean end-to-end latency was 813.1ms versus 770.8ms without evidence. In a complementary three-availability-zone AWS deployment, all 1,000 valid co-signed-DAG paths verified; issuance averaged 3.651ms and complete verification 5.015ms. The cloud result excludes TLS/mTLS, KMS, and model-serving latency.
VERA (Verifiable Edge Revocation for Agents), a verifier-checkable revocation contract and API emitted by agent-runtime adapters as signed evidence, is introduced and schema portability on A2A, AutoGen, and CrewAI artifacts is validated.
This work introduces the Counterfactual Fabrication Lab, a deterministic micro-lab where the correct action is known: do nothing, and presents the Counterfactual Fabrication Lab for measuring fabricated failures in self-improving agent harnesses.
Su Wang, Pinyan Qian, Yifan Lin et al.· arXiv.org· 6 citations· ⚡2
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
