An Empirical Benchmark of Small and Large Language Models for Extractive and Generative Question Answering with a Novel Confidence-Gated Hybrid Pipeline

Abstract

The rapid proliferation of transformer-based language models has produced two distinct deployment paradigms: Small Language Models (SLMs) optimized for computational efficiency, and Large Language Models (LLMs) designed for broad generalization. This paper presents a comprehensive empirical benchmark of four SLMs BERT-base, DistilBERT, ALBERT-base, and MobileBERT and four LLMs - Mistral-7B, LLaMA-2-7B, Gemma-2B, and GPT-3.5-turbo across extractive and generative Question Answering (QA) tasks evaluated on SQuAD 2.0, TriviaQA, and Natural Questions. Beyond standard accuracy metrics, we introduce four novel contributions: (1) a Hallucination Rate (HR) metric quantifying ungrounded LLM responses; (2) a Cost-Per-Correct-Answer (CPCA) metric enabling economic comparison across model classes; (3) a context length sensitivity analysis revealing systematic SLM degradation beyond 512 tokens; and (4) a Confidence-Gated QA (CG-QA) hybrid pipeline that intelligently routes queries to an SLM or LLM based on a calibrated confidence threshold, achieving 98.7% of GPT-3.5-turbo accuracy at 76% lower cost. Quantization experiments further demonstrate that INT4 LLMs reduce latency by approximately 6× with only 2.1% F1 degradation, substantially narrowing the SLM–LLM latency gap. A six-category failure mode taxonomy and a practical decision framework complete the study, providing actionable deployment guidance for NLP practitioners.