Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation datasets for AI models contribute to such advancement in AI. In the STEM domain, frontier models have consumed most of the available online data, creating the need for human-created datasets that codify the knowledge of leading experts in the domain. There are several STEM datasets available for the research community in this field. However, there are some gaps in these datasets, leaving room for improvement. Examples of gaps include (1) saturation in model performance on these datasets, leaving no head-room for meaningful evaluations, (2) skewed taxonomy distributions, (3) multiple choice question format that is misaligned with how scientists use AI in the real world, and (4) inaccurate answers and rationales partially led by a contest-based data collection and a time-bound review process. In this study, we present 'Expert-validated STEM QA', a high-quality, expert-validated STEM dataset (N=398) in Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. We (1) carefully designed a taxonomy with balanced distribution, (2) vetted question contributors with quality-driven incentive, (3) conducted multiple rounds of reviews with revisions validated by domain experts based on consensus, and (4) created the dataset in verifiable question and answer format. Our study demonstrated low performance ($<25\%$) of frontier AI models on the dataset as a benchmark. Post-training on a separate, private version of the dataset (N=2,000) increased performance of the open source model by $15\%$ relative to the baseline model (p=0.045) on the STEM subset of HLE-verified dataset, indicating potential utility of the dataset for model training. We have open-sourced a portion of our dataset for the AI research community.
Kihwan Han, Saurabh Patil, Chinmayee Shukla et al.· 0 citations
Large Language Model (LLM) agents have shown promise for automating data-science workflows, yet their end-to-end performance depends critically on the agent harness that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. Existing data-science agents often leave this harness implicit, making results difficult to reproduce, compare, and attribute across heterogeneous tasks. We introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation. DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation, and represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search. We further integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparison under a shared task interface, sandboxed runtime, and metric protocol. Experiments across agents, harnesses, models, and ablations show that explicit harness design improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. Our code is available at https://github.com/usail-hkust/dslighting
Artificial intelligence (AI) has fundamentally transformed online discourse, serving simultaneously as a source of and a potential solution to threats to democracy. This article conceptually examines AI’s multifaceted role in digital public spheres, beginning with an analysis of how AI shapes democratic processes online, ranging from information curation and political participation to dialogue facilitation and the construction of epistemic infrastructures. We identify several key risks: AI amplifies harmful content through automated content generation, coordinated manipulation, and the algorithmic mainstreaming of borderline content that often evades traditional detection systems. Within this broader democratic context, we position counter speech as a critical intervention strategy and examine both reactive measures (e.g., automated detection and content moderation) and proactive approaches (e.g., prebunking, algorithmic downranking, friction design, and AI-mediated dialogue). We argue that effective counter speech increasingly relies on human–AI collaboration, where AI supports human counter speakers with factual resources, emotional scaffolding, and scalability, while human actors preserve the authenticity and agency that make counter speech normatively meaningful. However, realizing the potential of such collaboration requires moving beyond technological fixes toward democratically legitimated governance structures. Drawing on examples from Wikipedia and decentralized platforms, we demonstrate how transparent and participatory institutional arrangements can foster more resilient discourse environments. We conclude that societies can harness AI’s democratic potential while mitigating its risks only by proactively establishing legitimate deliberative processes grounded in broadly shared discourse norms.
Diana Rieger, Mario Haim· Cogitatio (Cogitatio)· 0 citations
Modern industrial systems require increasing efficiency and flexibility to remain competitive. To achieve this, decision support systems are essential for assisting humans in steering manufacturing, helping operators meet productivity and quality goals while minimizing the environmental impact. Digital Twins (DTs) are a key enabling technology for these systems, as they can monitor the process and predict key performance indicators (KPIs) such as energy consumption and product quality across different scenarios. Conventionally, physics-based (PB) models are regarded as the gold standard for building DTs thanks to their interpretability and reliability. However, as industrial systems grow in complexity, modeling from first principles becomes prohibitive. Data-driven regression models offer a practical alternative by leveraging Machine Learning (ML) techniques to model the system directly from data. ML techniques can automatically capture general non-linear input-output relationships, which makes them significantly more flexible. Significant efforts have been made to digitalize factories within the Industry 4.0 paradigm, a prerequisite for training ML-based models. However, even with digitalization, a paradox arises: truly informative data remains scarce. Industries are typically risk-averse, limiting the exploration of new operating regimes to protect safety and maintain a stable production. As a result, historical data is typically confined to a small range and number of conditions, while the models must operate reliably in unseen scenarios to support human decision-making. Ironically, these are the scenarios where standard ML-based regression models lack trustworthiness, often lacking the robustness and physical consistency required to operate safely. This thesis therefore investigates how data-driven regression models can be made more trustworthy by design, by integrating prior knowledge and automatically selecting robust models, with a focus on building DTs for industrial machines. This research aligns with the EU Ethics Guidelines for Trustworthy Artificial Intelligence (AI) [55]. While these guidelines outline seven key requirements for trustworthiness, here the scope is limited to three pillars: human agency and oversight, technical robustness and safety, and transparency. Addressing these three pillars, the first part of the thesis focuses on integrating prior physical knowledge and human rationale into the modeling process. It starts by drawing theoretical connections between diverse practices in physics-informed ML, making it easier for practitioners to choose between methods. Specifically, it demonstrates that adding physical equations as input features is a special case of architectural inductive biases, and that data augmentation, with synthetic data generated prior to training, is a special case of soft constraints, the two becoming equivalent when the prior knowledge can be written explicitly as target values. It then proposes a framework designed to embed human decision-making rationale directly into the architecture and training process of models used for continuous manufacturing. Specifically, the system's time-series behavior is reduced to a static mapping. This preserves partial interpretability and aligns with operators' mental models, while enabling plug-and-play deployment over existing legacy control systems. To ensure physical consistency, the framework induces known monotonic relationships by regularizing the model's Jacobian matrix. It also emphasizes sensitivity to decision variables, a critical aspect often overlooked in conventional approaches. The framework is validated both on historical data and during a seven-month real-time deployment at a large-scale wooden fiberboard manufacturer. There, DTs were designed for five physically distinct process stages using the same approach, demonstrating its generality. The results show that the proposed framework enhances both model responsiveness and physical consistency compared to standard training, which often yields models with negligible sensitivity and limited practical utility. Extending the focus on technical robustness, the second part of the thesis complements the first by improving the robustness of data-driven regression models against distribution shifts by design. It addresses this challenge through a novel data splitting strategy named Leave-Boundary-Out (LBO). First, theoretical foundations are established, showing that out-of-distribution (OOD) regions can be exploited for more effective model selection. Because the sensitivity of the validation loss to hyperparameters is often amplified in these regions, sample-size requirements are reduced. The LBO algorithm then leverages this insight to select models with superior extrapolation performance. Validated on multiple synthetic and real-world benchmarks from diverse engineering domains, the results using LBO show that models tuned for extrapolation consistently outperform standard approaches in OOD scenarios. Importantly, for the Polynomial Lasso pipeline, models simpler than those tuned using standard approaches tend to extrapolate better under both the L0- and L1-norm complexity measures, with LBO acting as a regularization mechanism that favors OOD robustness over purely fitting in-distribution data; for SVR the trend was ambiguous. Additionally, LBO helps localized methods such as Radial Basis Function (RBF) kernel-based regressors learn more generalizable trends, which could prove valuable for optimization frameworks that largely rely on such models. Overall, this thesis provides theoretical, methodological, and experimental foundations demonstrating that trustworthy data-driven regression modeling requires tailored approaches based on use-case needs. Furthermore, it offers validated directions for achieving this in industrial and engineering contexts, such as DT development for process and product optimization.
Francisco Ambrosio Garcia· Lirias· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Consumer artificial intelligence chatbots are now accessed by hundreds of millions of users seeking health information, yet systematic evaluation of their safety boundary maintenance under real-world caregiver pressure remains scarce. We evaluated PediatricSafetyBench-v2, a benchmark of 600 pediatric health queries comprising 300 authentic caregiver queries sourced from the HealthCareMagic-100k-en physician consultation corpus and 300 matched adversarial variants incorporating six operationalized caregiver pressure patterns, across four consumer AI systems (GPT-4o-mini, Gemini-2.0-Flash, Claude-3.5-Haiku, and Llama-3.1-8B). Safety boundary maintenance was assessed using a validated five-component Safety Composite Score (maximum 15 points; safety-appropriate threshold of 10 or above), validated against independent human raters prior to full-corpus application (mean weighted kappa 0.76; Pearson r = 0.88). The overall safety-appropriate rate was 95.5%. Safety-oriented system prompt deployment improved safety-appropriate rates by 5.9 percentage points across all four models. Counter-intuitively, adversarial caregiver pressure was associated with higher rather than lower Safety Composite Score values for all four models across all ten topic categories and severity levels. False expertise claims were the most vulnerability-inducing pressure pattern, whereas emotional escalation was associated with the highest scores. Consumer AI systems maintain safety boundaries in the large majority of pediatric health interactions. PediatricSafetyBench-v2 is publicly released for longitudinal safety monitoring.
Vahid Zolfaghari· npj Digital Medicine· 0 citations
With the emergence of Physical AI, artificial intelligence is extending beyond screen-based applications to embodied systems that perceive, interact with, and act in the physical world. Unlike traditional AI, Physical AI operates under real-time safety constraints, continuously interacts with dynamic environments, and coexists with humans, introducing governance challenges that existing AI governance frameworks do not explicitly address. This paper presents a comprehensive survey of Physical AI governance from both scientific and operational perspectives. We synthesize existing governance principles and organize them into a unified governance framework tailored to physical AI systems. Building on this foundation, we propose a five-stage Physical AI lifecycle comprising research, design, data, model development, and deployment, and demonstrate how governance can be operationalized across each stage through concrete implementation practices. By connecting governance principles with engineering workflows, this survey provides a structured reference for researchers, developers, and policymakers to build Physical AI systems that are safe, trustworthy, and aligned with societal values.
Wang Yang, Shaojuan Wang, Hong-Xuan Liu et al.· 0 citations
Machine intelligence has conquered the symbolic world but stalled at the physical one. The stall is structural: physical AI faces a cold-start deadlock -- no intelligence without data, no data without deployed intelligence. Our thesis: the deadlock is real but unevenly distributed, and the exception has a name: the artificial physical world. Buildings, industrial facilities, and infrastructure are intentionally constituted and documented: designed artifacts ship with readable archives that precede and constitute their instances; here, norms are promulgated before instances, not averaged from them. Four contributions. (i) From a four-world ontology we derive a legitimacy criterion for constitutive prior frameworks: prior extraction is legitimate if and only if the object domain is intentionally constituted and has left a readable archive; the criterion is testable through direction of fit -- deviation from a constitutive norm is a violation in the world, not a revision of the model. (ii) We establish a layering lower bound: any such framework has at least four layers -- syntax, concept, knowledge, instance -- because four construction goals pair into mutually incompatible carriers. (iii) We register deployment claims across five industrial domains and a 32-class failure-mode vocabulary. (iv) We stake the framework on five falsifiable predictions, the central one checkable on the public engineering record: if it fails, the framework fails. Semi-formal arguments back these claims (Appendix A): a Gold-type boundary on rule coverage in archiveless worlds, a decidability result for failure reduction over closed concept layers, and a boundary theorem for certificate-anchored calculi. Large language models find an honored place here -- as readers of the archive, not as the archive. First of three companion works; the companions take up the questions deliberately left open.
Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and they lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor is defined as work a domain scientist would accept with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.
Aubrey M. Brueckner, D. Patel, Yuhuan He et al.· 1 citation
Large Language Models (LLMs) can generate natural language explanations that rationalize their own decisions, a phenomenon commonly referred to as self-explanations.Such explanations have emerged as a promising direction for explainable artificial intelligence (XAI), particularly for interpreting LLM behavior.However, while self-explanations often appear plausible, whether they faithfully reflect a model's underlying reasoning process remains an open question. In this opinion paper, we argue that self-explanations can be highly plausible, questionably faithful, and yet highly actionable. From a traditional XAI perspective, we identify the limitations of standard evaluation protocols for LLM-generated self-explanations and propose practical guidelines for assessing their plausibility and faithfulness. Moreover, we argue that evaluation should extend beyond these criteria to actionability, highlighting applications of LLM rationalization capabilities that support informed decision-making and appropriate action across diverse stakeholders.
E. Herrewijnen, Benedetta Muscato, Gizem Gezici et al.· 0 citations
Comprehensively evaluating AI agents across interactive environments is difficult due to fragmented tasks, scaffolds, verifiers, and scoring rules. Unfortunately, existing efforts to unify these evaluations are limited in scale and domain, making costly reruns necessary and leaving available data incomparable. We introduce MESSIER, a unified corpus of 957,611 records spanning 30 benchmarks, 745 agents, 11,891 tasks, and 74,263 verifiers. MESSIER combines public evaluation results with new runs on six underrepresented professional and scientific benchmarks, standardizing their heterogeneous components into a common schema. Using this corpus, we show that frontier progress is uneven across benchmark groups, with function-calling evaluations largely saturated, programming improving fastest, and enterprise workflows remaining most challenging. Counterfactual rescoring further shows that strict all-pass scoring in multi-verifier tasks can alter agent rankings. Finally, we derive capability scores from our corpus that correlate with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.84. The scores can also be estimated for subsets defined by domain, occupation, action space, or verifier type. In essence, MESSIER is a reusable resource for studying agent performance at scale, and a basis for designing better evaluations.
Stefan Krsteski, Charlotte Meyer, Guillaume Allegre et al.· 0 citations
Construction firms operate in knowledge-intensive and complex project environments where critical project knowledge is frequently fragmented across teams, documents, and digital systems. This fragmentation limits the systematic capture, structuring, and reuse of knowledge, which are fundamental processes for reducing knowledge loss, improving decision making, and enhancing project performance across organizational boundaries. Although effective knowledge mapping (KMp) is recognized as a valuable mechanism for organizing and disseminating such information, empirical evidence remains limited regarding how organizational, human, and technological factors interact to influence its effectiveness in construction contexts. This study addresses this gap by examining the interrelationships among these factors and their collective impact on KMp within construction firms. A quantitative, survey-based methodology was used to gather data from professionals within the construction industry. Structural equation modeling (SEM) using SPSS 23 and AMOS 24 was used to analyze the data and validate the measurement constructs and model fit. The study found that organizational frameworks, technological infrastructures, and human competencies significantly influence the effectiveness of KMp. Technological advancements were identified as a mediating factor, emphasizing the need for integration between digital tools and organizational culture to enhance knowledge-sharing processes. This study contributes to the knowledge management field by providing a systematic, data-driven perspective on the enablers of effective KMp. It extends the discourse on digital transformation and knowledge management, highlighting the importance of sociotechnical alignment between workforce capabilities and technological infrastructures. Future research should explore the role of artificial intelligence and machine learning in automating KMp processes. Practically, the findings provide construction firms with a structured strategy that integrates organizational alignment, workforce development, and technological investment to enhance KMp effectiveness and improve project decision making.
Safi Ullah, Xiaopeng Deng, Diana R. Anbar· Journal of construction engi...· 0 citations
A domain-specific legal artificial intelligence system for construction contract disputes via hybrid knowledge integration based on the retrieval-augmented generation (RAG) paradigm, integrating five core legal texts and 500 adjudication cases within a dual-engine architecture is proposed.
Ying Lu, Xin-Yun Shen, Yujing Wang et al.· Journal of construction engi...· 0 citations
