Aug 2026· Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
Abstract
Modern software systems rely on observability infrastructure (metrics, logs, and traces) to detect failures and maintain reliability. In practice, key observability decisions are made through untested defaults: static alerting thresholds, keyword-based log-level selection, and monitoring feature sets chosen by convention rather than evaluation. Despite growing interest in AIOps, no prior study has directly measured the gap between these default rule-based practices and machine learning across both anomaly detection and logging decisions. We present a two-part empirical study quantifying this gap. In Part A, we evaluate static-threshold baselines (𝜇 ± 3𝜎) against per-KPI machine learning models on the AIOps 2018 benchmark, which comprises 2.67 million labelled data points. ML models outperformed the static threshold on 91% of evaluable KPIs, though absolute performance varied widely (mean F1 = 0.41, median = 0.116), and an ablation study showed that 53% of monitoring features could be removed without degrading detection. SHAP analysis confirmed that no universal feature ranking exists across KPIs. In Part B, we mine 15,702 log statements from 15 open-source Node.js/TypeScript repositories and train classifiers to predict developer-chosen log levels from code context. The model achieves cross-project macro F1 of 0.92 when surrounding code (including existing log statements) is available, compared to 0.38 for a keyword heuristic. However, a level-name-stripped ablation reveals this drops to 0.52 without neighbouring log-level tokens, showing the dominant signal is inter-statement level clustering rather than deeper code-structural patterns. This clustering generalises across 15 independent codebases and 3 unseen test repositories. Our results demonstrate that default observability practices leave substantial performance on the table, while identifying boundary conditions where simple rules remain competitive.
MAGE explains how externalized knowledge, bounded action, independent evaluation, and retained human authority can compose into a governed engineering environment, and proposes tests of when that environment turns commodity intelligence into durable engineering progress.
James C. Davis, Kelechi G. Kalu, Huiyun Peng et al.· 1 citation
LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into the software supply chain. We make four contributions to this problem. First, we show that prior evaluation methodologies systematically inflate hallucination rates by misclassifying standard-library modules as hallucinations in some languages. For Python, the overestimation reaches 9.4 percentage points. Second, we evaluate seven inference-time defenses for mitigating package hallucinations, including five guided decoding strategies (Greedy, Contrastive, DoLa, Nudging, and Active Layer-Contrastive Decoding), an iterative self-refinement approach (Self-Refine), and a Retrieval-Augmented Generation (RAG)-based defense.. Across eight models spanning five families and four programming languages (Python, JavaScript, Ruby, Rust), RAG reduces the package hallucination rate (PHR) in 18 of 32 model--language configurations. Third, we introduce Package Utility (PU) to assess whether defenses preserve valid and task-relevant recommendations. Among strategies evaluated, Greedy decoding provides the strongest average mitigation--utility trade-off. Fourth, we stress-test all strategies under adversarial prompts seeded with fabricated package names and find that PHR surges by up to 45 percentage points relative to standard prompts, with Ruby consistently the most vulnerable language (80.9--95.2\%). Under adversarial conditions, RAG and Self-Refine outperform all decoding-only strategies, indicating that robust defense requires either external grounding or iterative self-verification when prompts are actively hostile. Our results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.
Albérick Euraste Djiré, Iyiola E. Olatunji, Melissa Tessa et al.· 1 citation
HawkEye is introduced, a modular, web-based vulnerability auditing platform designed to streamline security analysis by integrating multiple scanning tools within a unified dashboard and illustrates how consolidated reporting improves vulnerability prioritization for development teams.
D. R. Patil, Varad Salgare, Devaj Arya et al.· International Journal for Re...· 0 citations
By streamlining workflows and fostering collaboration, this platform offers a scalable, cost- effective solution for SMEs and contributes to software engineering by demonstrating how integrated technologies can modernize development processes in resource limited contexts, with potential for broader adoption in Albania and beyond.
OBJECTIVE
To prospectively evaluate whether modifying DSL version 5-based hearing aid (HA) fittings by adjusting gain on HA fitting software so that measured functional gain (FG) approached a one-third gain (1/3G) target could provide appropriate fitting outcomes in patients with sensorineural hearing loss.
METHODS
Twenty-four patients (48 ears) with bilateral sensorineural hearing loss underwent initial HA fitting using the DSL version 5 prescription formula. FG was measured at 250-4000 Hz, and HA gain was adjusted on HA fitting software so that FG approached the target 1/3 G. Speech discrimination scores at 65 and 80 dB SPL were evaluated after a two-week trial period using the 67-S Japanese monosyllable word list. Based on speech discrimination test results, ears were classified as well-fitting or non-well-fitting. FG values were compared between the two groups.
RESULTS
Twenty-one patients (42 ears) completed the study. Thirty-one ears (73%) were classified as well-fitting. Although HA gain was adjusted toward the target 1/3 G, measured FG values at 250 and 500 Hz remained lower than the target values. In well-fitting ears, low-frequency FG values were lower than the target 1/3 G, whereas FG at 2000 Hz was close to the target value. In contrast, non-well-fitting ears showed low-frequency FG values closer to the target 1/3 G, whereas FG values at 2000 and 4000 Hz remained below the target values.
CONCLUSIONS
Although HAs adjusted toward a 1/3 G target did not achieve the intended FG values, particularly at low frequencies, relatively favorable fitting outcomes were obtained in approximately three-quarters of the ears. In well-fitting ears, low-frequency FG remained below the target 1/3 G, whereas FG in the mid-frequency range around 2000 Hz was close to the target value. These findings provide a basis for future prospective studies to clarify how these FG characteristics should be applied to optimize HA adjustment.
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
