Full stack academic publishing in seizures with artificial intelligence: a cautionary tale

Abstract

Significance: There is potential for generative artificial intelligence (AI) to perform "full stack" academic publishing, including comprehensive manuscript fabrication, "peer"-review, and editorial decision making. This study evaluated the benefits and cautions in using AI for full stack academic publishing in clinical research about seizures. Methods: Three epileptologists each selected 1 published manuscript in Epilepsia and produced an AI-fabricated manuscript about seizures. AI tools were then used to perform "peer"-review and make editorial decisions with minimal human supervision. The rates of identifying AI-fabricated manuscripts and editorial decisions among the epileptologists was analyzed, and epileptologists' observations on how AI differed from typical human "peer"-review and editorial decisions was collected. Results: Four of 6 (67%) pairs of AI-fabricated and human-generated manuscripts were correctly identified. Only 1 AI-fabricated manuscript was editorially rejected, whereas another had a subtle indicator of fabrication. When compared to human content, AI generated content was comprehensive, detailed, and based on the highest standard of academic publication. However, generated content was more verbose, used more structured responses, and more critical feedback. Conclusions: These results demonstrate that the quality of AI-fabricated manuscripts about clinical research in seizures was sufficiently high that the current methods of peer and editorial review could not easily distinguish fabricated content compared to human-written content. Moreover, AI review and editorial decisions were high quality and appeared reasonable in many cases. AI tools have matured to the point that there should be thoughtful policies that permit AI assistance but prohibit insufficiently supervised autonomous scientific submissions, reviews, and editorial decisions.