EXPRESS: A network analytic comparison of word associations generated by humans and Large Language Models.
TL;DR
Analysis of word associations generated by humans and large language models indicate that latent semantic representations of human word associations are better integrated than that of the LLMs, illustrating how cross-comparisons with LLM-generated data can provide insights into the nature of human semantic representations.
Abstract
Word associations generated by humans and large language models (LLMs) were analyzed using a network science approach to provide insights into their semantic organization. Human data consisted of word associations from the "Small World of Words" project and from native speakers of Singapore English. LLM data were obtained from the "LLM World of Words" project where three LLMs (Haiku, Llama3, Mistral) were probed to produce word associations to a large set of cue words. Bipartite network projections onto a subset of cue word nodes enabled a comparison of identical-sized networks that also captured higher order associative patterns. Human and LLM networks had significantly different global network properties; specifically, human networks appeared to have more well-connected semantic organization than LLM networks, which were more sparsely connected. At the local node-level of the network, the two human networks were more similar to each other than to the LLM networks. Overall, the analyses indicate that latent semantic representations of human word associations are better integrated than that of the LLMs, illustrating how cross-comparisons with LLM-generated data can provide insights into the nature of human semantic representations.