Oct 2026· Big Data and Cognitive Computing· 0 citations· 11 references
TL;DR
This work proposes an automatic evaluation pipeline that enables large-scale, reproducible comparison of different configurations by analyzing generated answers and automating the scoring process, and identifies evaluators that best approximate human judgment.
Abstract
With the goal of facilitating efficient access to data to support the development of new R&D&I initiatives, this paper presents an information retrieval RAG-based system for projects funded by the Horizon 2020 program domain through multiple configurations and specific adaptations, considering CPU-only hardware demand. This work is an extension of a previous proposal in order to address the limitations when moving from exploratory prototypes to robust, large-scale experimentation. Additionally, we propose an automatic evaluation pipeline that enables large-scale, reproducible comparison of different configurations by analyzing generated answers and automating the scoring process. A methodology structured into three main phases guided the development of the system. First, project embeddings were generated using vector representation techniques, segmenting documents to compare models in terms of semantic quality. Second, relevant fragment retrieval was performed using language models to assess their relevance to a query. Third, system performance was evaluated using language model-based evaluators, automating the process. Additionally, an in-depth analysis identified evaluators that best approximate human judgment, comparing their results with metrics from the reference study. This work contributes to the State of the Art by proposing improvements in information retrieval systems.
A novel Retrieval-Augmented and LoRA-tuned Large Language Model is proposed for automated generation of structured software test cases from textual requirements and indicates the potential of retrieval augmentation and parameter-efficient fine-tuning for structured test-case generation within the investigated setting.
Sachin Shankarrao Ajmire, S. Thakare· Discover Computing· 0 citations
Extended abstract 3-063-26
Large language models (LLMs) provide a powerful means to search for and summarise content. However, their training and use is a rapidly growing driver of energy and resource use. In this presentation we describe the development of an LLM-based search tool for the UKRI Energy Demand Research C...
Thorin Daniel, Li-Rong Liu, Michael J. Fell et al.· Eceee ... summer study proce...· 0 citations
The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models.
This survey aims to systematically address resource efficiency challenges of Large Language Models by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs, and introduces a nuanced categorization of resource efficiency techniques by their specific resource types.
Preliminary results indicate that core categories of document mediation, such as relevance, completeness, citability and transparency, cannot be fully reduced to computational parameters but, instead, require continuous negotiation between automated models and disciplinary expertise.
Angelo La Gorga, Lorenzo Verna· JLIS.it· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.