Software testing often depends on a test oracle to determine whether or not an output is correct. However, many real-world systems lack reliable or affordable oracles. Metamorphic testing (MT) alleviates this problem by using metamorphic relations (MRs), which examine the correctness of relations between outputs produced from related inputs across multiple executions. While the effectiveness of MT depends heavily on the quality of MRs, existing MR evaluation is typically guided by informal or study-specific criteria, limiting standardization and repeatability, especially for complex and AI-integrated systems. This paper proposes a structured MR-evaluation rubric that operationalizes key criteria to support consistent and repeatable assessment. We conducted an empirical study on nine systems under test (SUTs) spanning three levels of complexity, using fresh sets of LLM-generated MRs for each SUT. We then compared evaluations from human experts against two widely used large language models (LLMs), namely ChatGPT and Gemini, by configuring these models as MR evaluators using the same rubric. The results show that human and LLM evaluators are more consistent when assessing simpler SUTs, but their judgments increasingly diverge as system complexity grows and evaluation criteria become more varied. The study further reveals clear differences in LLM-evaluation behaviors, indicating that evaluator choice can affect MR assessment outcomes in complex settings. Overall, the proposed rubric provides a standard framework for MR evaluation, and supports a systematic comparison between human and LLM evaluators, offering empirical evidence on when LLM-based MR evaluation is more reliable, and where additional care is needed as SUT complexity increases.
Yifan Zhang, Dave Towey, Matthew Pike et al.· Annual International Compute...· 0 citations
Evaluating the temporal consistency of RetrievalAugmented Generation (RAG) systems is challenging because conventional evaluation methods rely on ground-truth answers, which are difficult to define under continuously evolving knowledge. This paper presents a temporal metamorphic testing framework for RAG-based large language models (LLMs) that evaluates relational consistency across multiple executions over time rather than comparing outputs against static reference answers. To support temporal metamorphic testing, we construct controlled temporal revision families from seed facts extracted from real-world documents. Each fact is systematically cloned and transformed across multiple timestamps to simulate evolving knowledge states. Based on this design, temporal metamorphic relations are instantiated to systematically test RAG systems under different knowledge snapshots. Experiments on 2344 metamorphic test instances show that 48.04% of them violate the expected temporal relations, indicating that temporal inconsistencies are widespread in the evaluated RAG system. These results demonstrate that temporal metamorphic testing provides an effective way to assess temporal reliability in RAG systems without requiring explicit ground-truth answers.
Hongyu Wang, Dave Towey, Ge Zhang et al.· Annual International Compute...· 0 citations
Learning effective molecular representations is crucial for accurate property prediction in AI-aided drug discovery. However, most existing molecular pre-training methods are still primarily based on 2D topological graphs, limiting their ability to exploit 3D geometric information. Moreover, methods that do incorporate 3D geometry often do not distinguish between the roles of atom-centered and bond-centered representations. To address these limitations, we propose GDGraph, a geometryenhanced dual-view framework for molecular representation learning. GDGraph models molecular geometry from two complementary structural perspectives: an atom view for capturing global spatial dependencies and a bond view for modeling local geometric patterns. To support this dual-view design, we introduce a multi-scale geometric feature encoding scheme and a view-specific geometry-aware learning strategy, enabling each view to focus on the geometric dependencies it is best suited to capture. Extensive experiments demonstrate that GDGraph achieves strong and stable performance on molecular property prediction benchmarks, and effectively predicts geometrysensitive quantum chemical properties on the QM9 dataset.
Yu Liu, Jonathan D. Hirst, Jianfeng Ren et al.· Annual International Compute...· 0 citations