A Computational Stylometric Analysis of AI-Generated and Human-Written Short Stories in the Psychological Genre

Abstract

This study investigates whether the generative AI, especially, ChatGPT-4o can bring the same creativity as established American writer Karen Russell in short stories, particularly in psychological genre or not. For this purpose, two short stories of Karen Russell, “Reeling For The Empire” (2012), and the second story “St. Lucy’s Home For Girls Raised By Wolves” (2006), were selected. Two parallel short stories “Spinning the Ashes”, and “The Museum of Reclaimed Beasts” on the same themes were generated using ChatGPT-4o, for the comparison. Python based pipeline (NLTK, scikit-learn, VADER) was used in Google Collaboratory to extract the linguistic numerical profiles, and sentiment scores of each story. Linguistic numerical profile consisted of lexical features, syntactical features (part-of-speech tag distribution, adjectives per sentence), and sentiment polarity (VADER compound score). The findings reveal that the average sentence length in human-written short stories is significantly longer than the stories generated by AI, the hapax legomena was also more in human-written short stories. The adjectives in human-written short stories were almost double than AI-generated stories. The emotional/sentiment tone score was balanced in human-written short stories while the score was extreme and polarized in AI-generated short stories, which shows the emotional inconsistency of AI-generated short stories. These numerical profiles were interpreted according to the standard definition of Creativity by Runco and Jaeger (2012) which defines creativity as “originality + effectiveness”. According to Runco and Jaeger, creativity is unexpected and meaningful at the same time. ChatGPT-4o was able to write grammatically fluent stories with decent range of vocabulary but the patterns in it’s linguistic numerical profiles were not unexpected or meaningful in psychological genre. It was unable to develop nuanced emotional layering, originality, and effectiveness in it’s stories. It’s linguistic numerical profiles were significantly weaker in most of the areas in comparison of Karen Russell’s stories linguistic numerical profiles. It was unable to compete with the human-writer Karen Russell’s short stores in psychological genre. These findings have methodological and theoretical contribution in AI-assisted creative writing, computational literary studies, computational humanities, digital humanities, computational stylometry and LLM evaluation, AI-Human collaborative authorship.