Although LLMs significantly advance fact-checking, structural challenges persist, primarily hallucinations, data scarcity, data scarcity, and a lack of temporal awareness, and a research agenda prioritizing temporal-aware architectures, Small Language Models for sustainable deployment, and human-centered evaluation metrics are proposed.
Abstract
Automated fact-checking has become critical in combating misinformation, with Large Language Models (LLMs) emerging as powerful tools for this task. This systematic literature review analyzes how LLMs have been applied to automated fact-checking, identifying effective techniques, persistent challenges, and future directions. Following the PRISMA 2020 protocol, we analyzed 85 studies published between 2018 and 2025, extracted from five academic databases (IEEE Xplore, ACM Digital Library, Scopus, Web of Science, and ScienceDirect). We found that Prompt Engineering (84.7%), Retrieval-Augmented Generation (50.6%), and Fine-tuning (55.3%) dominate the literature. While proprietary models like GPT-4 achieve the highest performance (e.g., 88.5% F1-score), they do so at a significantly higher computational cost compared to classical Transformers. Critically, only 10.6% of the studies address temporality, despite the highly dynamic nature of real-world claims. Although LLMs significantly advance fact-checking, structural challenges persist, primarily hallucinations (48.2%), data scarcity (51.8%), and a lack of temporal awareness (10.6%). To overcome these barriers, we propose a research agenda prioritizing temporal-aware architectures, Small Language Models (SLMs) for sustainable deployment, and human-centered evaluation metrics.
Adopting Continuous Integration (CI) often requires writing YAML configurations that are error-prone and challenging to maintain. Despite increasing LLM use in software engineering, their ability to generate CI configurations from natural language across services and model families remains unclear. This paper presents a large empirical study on using LLMs to generate CI configurations. We introduce DOC2CI, a benchmark of 3,363 description-to-YAML pairs collected from the official documentation of four CI services, and evaluate 14 open-weight models from 7B-34B parameters together with GPT-4o and GPT-4.1, producing over 53,000 configurations. We assess both reference alignment and schema validity to determine whether the generated configurations are structurally valid. We further develop a failure taxonomy from a manual analysis of 385 configurations and examine why LLMs disagree. Across models and services, exact reference reproduction never exceeds 3.1%, and while 97% of outputs parse as YAML, only 71% satisfy service schemas. Larger models improve structural validity, but code specialization provides no consistent advantage over comparable general models. Model differences are driven largely by output completeness: for the same request, some models generate the expected fragment while others produce a full workflow. Finally, a training-free schema-guided repair method improves schema validity to 94%, while fine-tuning improves similarity to documentation but reduces standalone validity. This suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.
Large language models (LLMs) often rely on shortcuts rather than systematic reasoning, raising safety concerns in medical applications. Allowing models to abstain when uncertain improves reliability but introduces a coverage accuracy tradeoff. We propose a two-stage framework for medical hypothesis verification in multiple-choice settings that manages this tradeoff through targeted ontology grounding, applied only when the model abstains. We show that abstention is not random but reflects genuine uncertainty, with abstained predictions associated with lower confidence. Across two frontier models (GPT-5.5, accessed via the Azure OpenAI API, and DeepSeek-R1), the proposed framework improves question-level accuracy by 9.6 percentage points (82.9% to 92.5%) and hypothesis-level accuracy by 4.2 percentage points (92.0% to 96.2%). Our experiments conducted on MedReason and MedQA show that abstention can be repurposed as a control signal for selective reasoning refinement, achieving knowledge-graph-level performance without explicit knowledge graph construction.
Uma Ranjan, Kunal Tilaganji, Aditya Koul et al.· 0 citations
Property-Based Testing (PBT) automatically generates test inputs to validate properties of programs, shifting developers’ effort from writing examples to specifying invariants. While the technique has gained popularity in Python through the Hypothesis framework, little is known about how developers adopt and use it in practice. This paper reports on three empirical studies. First, we analyzed 367 PBTs from 244 Python projects, classifying them into nine property categories and quantifying their use of Hypothesis constructs. We found that Test Oracle properties dominate (29.97%), and that PBTs are generally concise (median 14 LOC), relying heavily on built-in strategies (75.20%), but also on external (22.62%) and internal (17.17%) ones. Second, we studied 213 Stack Overflow posts tagged with PBT, revealing that the main challenges developers face concern data generation strategies (36.62%), especially for composite and tabular data (24.36%). Finally, we evaluated Ghostwriter, Hypothesis’s automated test generator, against 203 tests from our dataset; only 18.23% were fully automatable, while most required partial adaptation (30.05%) or were incompatible (51.72%). Together, our findings provide the largest empirical characterization of PBT in Python to date, highlight developers’ difficulties in adopting the technique, and expose limitations of current tool support.
Isadora Campos De Oliveira, Arthur Lisboa Corgozinho, Henrique Rocha et al.· Empirical Software Engineeri...· 0 citations