Are Large Language Models Consistent Reasoners?

Abstract

This paper investigates the reasoning capabilities of Large Language Models (LLMs) by adopting a problem-solving approach. For maritime situation awareness, ChatGPT-4 was tested on a practical task for ship identification by gradually integrating a variety of information sources and observations. Experiments were conducted by using data of variable types and quality and then measuring the consistency of results when several iterations are performed and similar prompts are used. The goal is also to test whether LLMs can support decision making and can be used in daily activities of security stakeholders. In contrast to accuracy analysis, where comparison to ground truth is available, this approach only covers comparisons on LLMs results in an effort to emphasize discrepancies. The experiments corroborate the fact that LLMs are not consistent in their responses and their outcomes should be carefully examined before being considered for operational tasks. These findings offer deeper insights on LLMs reasoning capabilities and a new perspective for more nuanced utilization of LLM models for data fusion problem-solving tasks.