Large language model (LLM) agents are increasingly embedded in scientific workflows for literature analysis, drafting, and review. Existing systems advance autonomous discovery and manuscript generation, but do not resolve the governance problem that arises when ideas, methods, results, and claims propagate through AI-assisted workflows without mandatory human approval or artifact-level traceability. This paper proposes Paper Pilot, a human-in-the-loop expert system for evidence-traceable scientific manuscript generation in applied sciences. It adapts the Collaborative Agent Reasoning Engineering (CARE) methodology to manuscript development through manuscript-owner approval gates, explicit no-pass criteria, claim classification, audit logging, advisory LLM review, and evidence-locked revision control. The framework defines eight approval gates across the idea-to-claim pipeline and distinguishes literature-grounded from artifact-grounded claims, requiring reported numbers and interpretations to remain traceable to approved evidence; its system prompt is openly released for deployment in ChatGPT, Gemini, Claude, or institutional LLM environments. As a first empirical validation, we evaluate the citation-grounding layer with a controlled, mechanically scored benchmark (two commercial LLMs, real arXiv papers, no LLM judge): under coverage pressure ungated drafters fabricated up to 25% of their citations and never flagged an evidence gap, whereas the same models under Paper Pilot's evidence-locked rules produced zero fabricated citations and surfaced the planted gaps as explicit placeholders. Preliminary results for result grounding, revision, and adversarial robustness point the same way; full evaluation is left to future work. Paper Pilot positions LLM-assisted writing as a controlled human-AI decision-support process rather than a fully autonomous authorship pipeline.
Conversational analytics systems assume the user already has a well-formed question, leaving a non-expert facing a blank query box on an unfamiliar enterprise schema. Commercial 'proactive' tools narrow this gap only by detecting statistical anomalies over analyst-curated metric layers, and academic next-question recommenders depend on query logs that a fresh dataset lacks. We describe a production analytics system that inverts the interaction model from question-first to analyst-first through two coupled architectural ideas. First, a pluggable domain-expert 'skill' abstraction: a folder-based, database-free subject-matter pack (a manifest, per-stage prompt facets, keyword-routed references, report templates, and optional compute) auto-selected per (client, dataset) by deterministic schema matching and spliced as a cross-cutting concern into every stage of an agentic pipeline, the schema explorer, and the report engines, degrading to a strict no-op when absent. Because a skill is a self-contained folder resolved deterministically, the catalogue is open-ended: an extensible marketplace of domain experts. Second, an offline knowledge-compilation loop: an agent probes the dataset's parquet via DuckDB (zero load on production), runs critic-gated per-table convergence with self-healing retries, and data-validates joins by value overlap, producing durable schema knowledge that drives standing expert reports whose every published metric is re-verified by re-executing its evidence SQL, plus suggested questions that mirror the report agenda. These close a proactive loop: reports surface numbers, the numbers seed questions, and a click launches a verified deep dive, all before the query box is used. We give a formal model and report illustrative single-tenant evidence. We make no user-study or benchmark claims; the contribution is the architecture and its defensibility.
With the increasing development of AI regulatory frameworks, ensuring that artificial intelligence systems, particularly generative models, operate in accordance with legal and ethical standards has become a critical priority. Existing proposals for AI alignment and value-guided behavior, however, face some limitations. Approaches such as Constitutional AI depend on human supervision, while broad normative frameworks like the Good-for-Humanity (GfH) principle may be overly general and ambiguous to provide actionable governance guidance. To overcome these limitations, we propose a hybrid approach called Statutory AI that employs pre-existing human-authored principles drawn from specific themes within a legal corpus. Specifically, Statutory AI uses legal texts as a constitutional framework, enabling AI systems to autonomously critique and revise their outputs according to established norms. It operates in two stages, both using Chain-of-Thought prompting. The first stage classifies the user prompt into one of the identified themes, while the second stage analyzes it in conjunction with relevant articles selected from the legal corpus of that theme. To illustrate the potential of our approach, we conducted an experiment involving 1,000 red-teaming prompts and five penal themes: discrimination, disclosure of confidential information, violence, fraud, and abuse of vulnerable persons. Statutory AI reduced harmful content by 52 to 59 percentage points across tested models, approximately 10 percentage points higher than standard Constitutional AI, while cutting computation time by over 50%.
Cindy Delage, St\'ephane Canu, Marc D\'ecombas et al.· 0 citations
Large language models (LLMs) achieve high scores on medical knowledge examinations, yet real-world oncology is not a knowledge test--it is a sequence of guideline-pathway choices, escalation judgments, and commitments under uncertainty. Existing benchmarks largely measure factual recall, leaving open whether frontier LLMs share decision-path blind spots that combining models cannot fix. We built the Oncology Decision Boundary Benchmark (ODBB)--2,005 oncology decision points across NCCN guidelines and colorectal cancer cases--and evaluated nine frontier LLMs (four closed-source, five open-weight families) released between June 2025 and April 2026. A fully deterministic scorer (zero LLM inference) classified outputs into 14 failure types, independently validated by two oncologists (Cohen's weighted $\kappa$ = 0.939 and 0.790) on a 225-item stratified sample. Treating the nine as a pooled super-model, 42.1% (Wilson 95% CI 40.0--44.3%) of all items--35.7% of the 1,586 NCCN items and 66.4% of the 419 colorectal-cancer cases--were answered correctly by none, with failures concentrated in choosing between guideline pathways before reasoning within any: a consistent blind spot in clinical meta-judgment that likely requires architectural intervention rather than more training data. Two models tuned for decisiveness (GPT-5.5, Gemini 3.1 Pro Preview) made unsafe commitments three to five times more often than the seven cautious models without scoring higher. In 3--9% of items, models stated the correct next clinical step yet did not commit to it--failures of decision, not knowledge. Model quality is no longer the primary bottleneck for clinical LLM deployment; the binding constraint is the assumption that any single model can be the sole basis for a clinical decision. Progress requires architectures that detect when a model reaches its competence boundary and route the decision to a clinician.
Zhang Sheng, Jinming Li, Wangyang Chen et al.· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation datasets for AI models contribute to such advancement in AI. In the STEM domain, frontier models have consumed most of the available online data, creating the need for human-created datasets that codify the knowledge of leading experts in the domain. There are several STEM datasets available for the research community in this field. However, there are some gaps in these datasets, leaving room for improvement. Examples of gaps include (1) saturation in model performance on these datasets, leaving no head-room for meaningful evaluations, (2) skewed taxonomy distributions, (3) multiple choice question format that is misaligned with how scientists use AI in the real world, and (4) inaccurate answers and rationales partially led by a contest-based data collection and a time-bound review process. In this study, we present 'Expert-validated STEM QA', a high-quality, expert-validated STEM dataset (N=398) in Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. We (1) carefully designed a taxonomy with balanced distribution, (2) vetted question contributors with quality-driven incentive, (3) conducted multiple rounds of reviews with revisions validated by domain experts based on consensus, and (4) created the dataset in verifiable question and answer format. Our study demonstrated low performance ($<25\%$) of frontier AI models on the dataset as a benchmark. Post-training on a separate, private version of the dataset (N=2,000) increased performance of the open source model by $15\%$ relative to the baseline model (p=0.045) on the STEM subset of HLE-verified dataset, indicating potential utility of the dataset for model training. We have open-sourced a portion of our dataset for the AI research community.
Kihwan Han, Saurabh Patil, Chinmayee Shukla et al.· 0 citations
Large Language Model (LLM) agents have shown promise for automating data-science workflows, yet their end-to-end performance depends critically on the agent harness that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. Existing data-science agents often leave this harness implicit, making results difficult to reproduce, compare, and attribute across heterogeneous tasks. We introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation. DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation, and represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search. We further integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparison under a shared task interface, sandboxed runtime, and metric protocol. Experiments across agents, harnesses, models, and ablations show that explicit harness design improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. Our code is available at https://github.com/usail-hkust/dslighting
Artificial intelligence (AI) has fundamentally transformed online discourse, serving simultaneously as a source of and a potential solution to threats to democracy. This article conceptually examines AI’s multifaceted role in digital public spheres, beginning with an analysis of how AI shapes democratic processes online, ranging from information curation and political participation to dialogue facilitation and the construction of epistemic infrastructures. We identify several key risks: AI amplifies harmful content through automated content generation, coordinated manipulation, and the algorithmic mainstreaming of borderline content that often evades traditional detection systems. Within this broader democratic context, we position counter speech as a critical intervention strategy and examine both reactive measures (e.g., automated detection and content moderation) and proactive approaches (e.g., prebunking, algorithmic downranking, friction design, and AI-mediated dialogue). We argue that effective counter speech increasingly relies on human–AI collaboration, where AI supports human counter speakers with factual resources, emotional scaffolding, and scalability, while human actors preserve the authenticity and agency that make counter speech normatively meaningful. However, realizing the potential of such collaboration requires moving beyond technological fixes toward democratically legitimated governance structures. Drawing on examples from Wikipedia and decentralized platforms, we demonstrate how transparent and participatory institutional arrangements can foster more resilient discourse environments. We conclude that societies can harness AI’s democratic potential while mitigating its risks only by proactively establishing legitimate deliberative processes grounded in broadly shared discourse norms.
Diana Rieger, Mario Haim· Cogitatio (Cogitatio)· 0 citations
Modern industrial systems require increasing efficiency and flexibility to remain competitive. To achieve this, decision support systems are essential for assisting humans in steering manufacturing, helping operators meet productivity and quality goals while minimizing the environmental impact. Digital Twins (DTs) are a key enabling technology for these systems, as they can monitor the process and predict key performance indicators (KPIs) such as energy consumption and product quality across different scenarios. Conventionally, physics-based (PB) models are regarded as the gold standard for building DTs thanks to their interpretability and reliability. However, as industrial systems grow in complexity, modeling from first principles becomes prohibitive. Data-driven regression models offer a practical alternative by leveraging Machine Learning (ML) techniques to model the system directly from data. ML techniques can automatically capture general non-linear input-output relationships, which makes them significantly more flexible. Significant efforts have been made to digitalize factories within the Industry 4.0 paradigm, a prerequisite for training ML-based models. However, even with digitalization, a paradox arises: truly informative data remains scarce. Industries are typically risk-averse, limiting the exploration of new operating regimes to protect safety and maintain a stable production. As a result, historical data is typically confined to a small range and number of conditions, while the models must operate reliably in unseen scenarios to support human decision-making. Ironically, these are the scenarios where standard ML-based regression models lack trustworthiness, often lacking the robustness and physical consistency required to operate safely. This thesis therefore investigates how data-driven regression models can be made more trustworthy by design, by integrating prior knowledge and automatically selecting robust models, with a focus on building DTs for industrial machines. This research aligns with the EU Ethics Guidelines for Trustworthy Artificial Intelligence (AI) [55]. While these guidelines outline seven key requirements for trustworthiness, here the scope is limited to three pillars: human agency and oversight, technical robustness and safety, and transparency. Addressing these three pillars, the first part of the thesis focuses on integrating prior physical knowledge and human rationale into the modeling process. It starts by drawing theoretical connections between diverse practices in physics-informed ML, making it easier for practitioners to choose between methods. Specifically, it demonstrates that adding physical equations as input features is a special case of architectural inductive biases, and that data augmentation, with synthetic data generated prior to training, is a special case of soft constraints, the two becoming equivalent when the prior knowledge can be written explicitly as target values. It then proposes a framework designed to embed human decision-making rationale directly into the architecture and training process of models used for continuous manufacturing. Specifically, the system's time-series behavior is reduced to a static mapping. This preserves partial interpretability and aligns with operators' mental models, while enabling plug-and-play deployment over existing legacy control systems. To ensure physical consistency, the framework induces known monotonic relationships by regularizing the model's Jacobian matrix. It also emphasizes sensitivity to decision variables, a critical aspect often overlooked in conventional approaches. The framework is validated both on historical data and during a seven-month real-time deployment at a large-scale wooden fiberboard manufacturer. There, DTs were designed for five physically distinct process stages using the same approach, demonstrating its generality. The results show that the proposed framework enhances both model responsiveness and physical consistency compared to standard training, which often yields models with negligible sensitivity and limited practical utility. Extending the focus on technical robustness, the second part of the thesis complements the first by improving the robustness of data-driven regression models against distribution shifts by design. It addresses this challenge through a novel data splitting strategy named Leave-Boundary-Out (LBO). First, theoretical foundations are established, showing that out-of-distribution (OOD) regions can be exploited for more effective model selection. Because the sensitivity of the validation loss to hyperparameters is often amplified in these regions, sample-size requirements are reduced. The LBO algorithm then leverages this insight to select models with superior extrapolation performance. Validated on multiple synthetic and real-world benchmarks from diverse engineering domains, the results using LBO show that models tuned for extrapolation consistently outperform standard approaches in OOD scenarios. Importantly, for the Polynomial Lasso pipeline, models simpler than those tuned using standard approaches tend to extrapolate better under both the L0- and L1-norm complexity measures, with LBO acting as a regularization mechanism that favors OOD robustness over purely fitting in-distribution data; for SVR the trend was ambiguous. Additionally, LBO helps localized methods such as Radial Basis Function (RBF) kernel-based regressors learn more generalizable trends, which could prove valuable for optimization frameworks that largely rely on such models. Overall, this thesis provides theoretical, methodological, and experimental foundations demonstrating that trustworthy data-driven regression modeling requires tailored approaches based on use-case needs. Furthermore, it offers validated directions for achieving this in industrial and engineering contexts, such as DT development for process and product optimization.
Machine intelligence has conquered the symbolic world but stalled at the physical one. The stall is structural: physical AI faces a cold-start deadlock -- no intelligence without data, no data without deployed intelligence. Our thesis: the deadlock is real but unevenly distributed, and the exception has a name: the artificial physical world. Buildings, industrial facilities, and infrastructure are intentionally constituted and documented: designed artifacts ship with readable archives that precede and constitute their instances; here, norms are promulgated before instances, not averaged from them. Four contributions. (i) From a four-world ontology we derive a legitimacy criterion for constitutive prior frameworks: prior extraction is legitimate if and only if the object domain is intentionally constituted and has left a readable archive; the criterion is testable through direction of fit -- deviation from a constitutive norm is a violation in the world, not a revision of the model. (ii) We establish a layering lower bound: any such framework has at least four layers -- syntax, concept, knowledge, instance -- because four construction goals pair into mutually incompatible carriers. (iii) We register deployment claims across five industrial domains and a 32-class failure-mode vocabulary. (iv) We stake the framework on five falsifiable predictions, the central one checkable on the public engineering record: if it fails, the framework fails. Semi-formal arguments back these claims (Appendix A): a Gold-type boundary on rule coverage in archiveless worlds, a decidability result for failure reduction over closed concept layers, and a boundary theorem for certificate-anchored calculi. Large language models find an honored place here -- as readers of the archive, not as the archive. First of three companion works; the companions take up the questions deliberately left open.
Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with"function calling"saturated,"programming"improving the fastest, and"enterprise workflows"remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.
Stefan Krsteski, Charlotte Meyer, Guillaume Allegre et al.· 0 citations
Construction firms operate in knowledge-intensive and complex project environments where critical project knowledge is frequently fragmented across teams, documents, and digital systems. This fragmentation limits the systematic capture, structuring, and reuse of knowledge, which are fundamental processes for reducing knowledge loss, improving decision making, and enhancing project performance across organizational boundaries. Although effective knowledge mapping (KMp) is recognized as a valuable mechanism for organizing and disseminating such information, empirical evidence remains limited regarding how organizational, human, and technological factors interact to influence its effectiveness in construction contexts. This study addresses this gap by examining the interrelationships among these factors and their collective impact on KMp within construction firms. A quantitative, survey-based methodology was used to gather data from professionals within the construction industry. Structural equation modeling (SEM) using SPSS 23 and AMOS 24 was used to analyze the data and validate the measurement constructs and model fit. The study found that organizational frameworks, technological infrastructures, and human competencies significantly influence the effectiveness of KMp. Technological advancements were identified as a mediating factor, emphasizing the need for integration between digital tools and organizational culture to enhance knowledge-sharing processes. This study contributes to the knowledge management field by providing a systematic, data-driven perspective on the enablers of effective KMp. It extends the discourse on digital transformation and knowledge management, highlighting the importance of sociotechnical alignment between workforce capabilities and technological infrastructures. Future research should explore the role of artificial intelligence and machine learning in automating KMp processes. Practically, the findings provide construction firms with a structured strategy that integrates organizational alignment, workforce development, and technological investment to enhance KMp effectiveness and improve project decision making.
Safi Ullah, Xiaopeng Deng, Diana R. Anbar· Journal of construction engi...· 0 citations
A domain-specific legal artificial intelligence system for construction contract disputes via hybrid knowledge integration based on the retrieval-augmented generation (RAG) paradigm, integrating five core legal texts and 500 adjudication cases within a dual-engine architecture is proposed.
Ying Lu, Xin-Yun Shen, Yujing Wang et al.· Journal of construction engi...· 0 citations
