Audience-Customized Translation of Rule-Based Evidence with Large Language Models Across Multiplexer Benchmarks

Abstract

Evolutionary rule-based machine learning (ERBML) algorithms can capture complex relationships while still yielding highly interpretable models comprised of IF:THEN rules. During prediction, 'matching' rules contribute to, and form the explanation for, the model's prediction. However, rules and their associated parameters (in their raw form) are likely too technical for their intended users. This study examines the feasibility of using a large language model (LLM) to translate the prediction evidence from matching rules into natural language text for different audiences, e.g. layman, clinician, expert. Using models trained by the 'HEROS' ERBML on MUX benchmarks, we evaluate LLM text quality metrics under different scenarios (i.e. 1,800 prediction explanations). We observe that (1), intuitively, LLM quality performance improves on HEROS models that have been more ideally trained, (2) making a glossary available to the LLM to define feature names generally raises explanation audience-fit scores and sometimes lowers overstatement (beyond rule-evidence), but it also lengthens explanations and often increases hallucination rate, and (3) audience customization creates some LLM performance trade-offs. These results suggest that constrained LLM translation of rules for natural language prediction explanations is feasible, while highlighting the importance of carefully designing the LLM prompts and evidence input from the ERBML.