Graph Neural Networks with Code Structure Embeddings for Model-Agnostic Multi-Class Detection of AI-Generated Programs

Abstract

Most detectors of AI-generated code answer a binary question: human or machine. We study the multi-class problem, naming the generator family behind a program, and the open-set problem that follows from it, in which the generator was never seen during training. Our detector, GraphCSE, represents a program as a heterogeneous code graph with four edge relations and fuses structural (syntax-type) and semantic (token) node channels through a learned per-node gate before relation-aware graph attention. On a controlled synthetic corpus of 2,500 Python programs spanning a human class built from eight style archetypes and four generator-family proxies, GraphCSE reaches 92.1 ± 0.6% macro-F1, the strongest learned model we test and 11.2 points above a GCN on identical graphs, though character n-gram TF-IDF remains slightly ahead in-distribution (93.4 ± 0.5%). The paper's central findings concern unseen generators. First, a negative one: on single programs, all five rejection rules we test (closed-set argmax, centroid and k-NN distance, energy, maximum softmax probability) fail on held-out families, with mean AUROC no better than 0.65: a tight unseen style inside the human envelope is indistinguishable from an unusual human. Second, a constructive one: judging a set of programs from the same source by a two-sided anomaly test on embedding statistics, calibrated only on human data, restores model-agnostic detection, reaching 86.4-97.3% mean detection at ten programs per source (AUROC up to 0.972) in the graph embeddings versus 53.6% in raw TF-IDF space. Provenance of unseen generators is a property of collections, not snippets.