CustomNerd: A Framework and Tool for Fast Deployment of Production-Ready Expertise-Based Question Answering Systems
TL;DR
This paper describes how the CustomNerd system can be used to build an expertise-driven application through a running example, explains the system’s architecture, discusses related work, and evaluates CloudNerd on a 120-question Stack Overflow benchmark against OpenDeepResearch and AutoGPT using semantic textual similarity and RAGAS metrics with paired permutation tests.
Abstract
CustomNerd is a modular framework to build domain-specific expertise systems using a common set of core components. The system is designed to support multiple domains and has already been deployed to instantiate expertise-driven systems in nutrition (DietNerd), news (NewsNerd), cloud technology (CloudNerd), and space science (SpaceNerd). CustomNerd incorporates specialized knowledge from domain-specific data sources that a user provides and processes the results of searches on those sources. The system consists of five main components: (i) Query Conversion, which converts user input into structured queries for one or more application-specific databases of validated articles; (ii) Extraction, which uses the structured queries to extract relevant data from those external databases; (iii) Filtering and Classification, which identifies and prioritizes relevant documents among those retrieved; (iv) Reliability Assessment, which evaluates the quality of the documents (e.g., statistical validity, absence of conflict of interest); and (v) Response Generation, which produces text outputs based on data sources considered to be relevant and reliable. In addition, the system can (i) generate questionnaires based on a user query, (ii) allow users to upload their own documents, and (iii) work with a variety of Large Language Model (LLM) engines. This paper describes how the system can be used to build an expertise-driven application through a running example, explains the system’s architecture, discusses related work, and evaluates CloudNerd on a 120-question Stack Overflow benchmark against OpenDeepResearch and AutoGPT using semantic textual similarity and RAGAS metrics with paired permutation tests. The results show that the RAG frameworks of CustomNerd and OpenDeepResearch both achieve better semantic fidelity than the agentic AutoGPT. Further, CustomNerd and OpenDeepResearch show complementary semantic advantages. Further, simply concatenating the two answers yields semantic benefits compared to the outputs of either one alone.