Skip to content
Open access

Implementation and Evaluation of an Efficient RAG System in Resource-Constrained Environments

Oct 2026 · Big Data and Cognitive Computing · 0 citations · 11 references

TL;DR

This work proposes an automatic evaluation pipeline that enables large-scale, reproducible comparison of different configurations by analyzing generated answers and automating the scoring process, and identifies evaluators that best approximate human judgment.

Abstract

With the goal of facilitating efficient access to data to support the development of new R&D&I initiatives, this paper presents an information retrieval RAG-based system for projects funded by the Horizon 2020 program domain through multiple configurations and specific adaptations, considering CPU-only hardware demand. This work is an extension of a previous proposal in order to address the limitations when moving from exploratory prototypes to robust, large-scale experimentation. Additionally, we propose an automatic evaluation pipeline that enables large-scale, reproducible comparison of different configurations by analyzing generated answers and automating the scoring process. A methodology structured into three main phases guided the development of the system. First, project embeddings were generated using vector representation techniques, segmenting documents to compare models in terms of semantic quality. Second, relevant fragment retrieval was performed using language models to assess their relevance to a query. Third, system performance was evaluated using language model-based evaluators, automating the process. Additionally, an in-depth analysis identified evaluators that best approximate human judgment, comparing their results with metrics from the reference study. This work contributes to the State of the Art by proposing improvements in information retrieval systems.

Read PDF

Similar papers

#large language models Open access Oct 2026

A retrieval-augmented LoRA-Tuned LLM framework for structured software test case generation

A novel Retrieval-Augmented and LoRA-tuned Large Language Model is proposed for automated generation of structured software test cases from textual requirements and indicates the potential of retrieval augmentation and parameter-efficient fine-tuning for structured test-case generation within the investigated setting.

Sachin Shankarrao Ajmire, S. Thakare · 0 citations
#small language model Open access Oct 2026

Pragmatic AI: building low-impact LLM tools for energy research impact

Extended abstract 3-063-26 Large language models (LLMs) provide a powerful means to search for and summarise content. However, their training and use is a rapidly growing driver of energy and resource use. In this presentation we describe the development of an LLM-based search tool for the UKRI Energy Demand Research C...

Thorin Daniel, Li-Rong Liu, Michael J. Fell et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models.

N. Hart, Nicolas Weeger · 0 citations
Review Open access Jan 2024

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

This survey aims to systematically address resource efficiency challenges of Large Language Models by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs, and introduces a nuanced categorization of resource efficiency techniques by their specific resource types.

Guangji Bai, Zheng Chai, Chen Ling et al. · 120 citations · ⚡5
Open access Sep 2026

Evaluating Retrieval-Augmented Generation for personal collections: architecture, models and criteria

Preliminary results indicate that core categories of document mediation, such as relevance, completeness, citability and transparency, cannot be fully reduced to computational parameters but, instead, require continuous negotiation between automated models and disciplinary expertise.

Angelo La Gorga, Lorenzo Verna · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.