Explainable Large Language Model for Intelligent Decision Support in Complex Domain

Abstract

Despite showing great skills in working with unstructured data and generating appropriate answers, large language models (LLMs) cannot be applied in critical settings due to their lack of transparency and susceptibility to hallucination. The paper describes an approach to creating explainable LLM-based decision support systems, which includes breaking down complex questions into small subtasks that are processed via auditable pipelines. The approach employs retrieval-augmented generation along with domain-specific knowledge bases and computational instruments for the purpose of producing explanations and demonstrating intermediate reasoning steps while emphasizing data limitations. The approach is tested on clinical decision support problems for a multiple myeloma population. Comparing the results with those obtained when using monolithic LLMs as well as explainable machine learning approaches proves that structured decomposition leads to better decision traceability while achieving similar performance.