Tradespace Analysis using GPT: A Comparative Study with Humans
TL;DR
This study compares GPT-4's performance with that of 30 human participants from diverse backgrounds and shows that the CS/AI group had the most closely aligned responses with GPT-4.
Abstract
With large language models (LLMs) becoming ever more popular and their usage expanding into various domains, this study explores how effective an LLM like GPT-4 would be in analyzing requirement specification documents and using relevant information from those to create a tradespace matrix. This study compares GPT-4's performance with that of 30 human participants from diverse backgrounds, categorized into three groups: engineering, non-technical and (CS/AI). Each participant and the LLM completed a survey based on materials provided for evaluating a complex system design and populating a trade-space matrix. The analysis of these responses included within-group heatmaps, across-group comparisons, and human-versus-GPT-4 heatmap evaluations. The results show that the CS/AI group had the most closely aligned responses with GPT-4. This study demonstrates how LLMs can augment trade-space exploration and serve as an alternative in cases where employing a team of human experts from multiple backgrounds is not feasible.