Leveraging Transformer Embeddings for Real-Time Discovery of Malicious URL Campaigns in the Generative-AI Threat Era

Abstract

The proliferation of algorithmically generated malicious URLs presents a critical challenge for modern cybersecurity, requiring a shift from syntactic pattern-matching toward semantic understanding grounded in pre-trained transformer foundations. Building on our prior work on billion-scale semantic search and density-based campaign clustering, this paper presents a unified, deployable framework for real-time campaign discovery. The framework converts raw URL streams into dense Sentence-BERT embeddings and couples approximate nearest neighbor search with online density-based clustering, discovering emerging campaigns without prior knowledge of their number or shape. Our central finding is that the choice of semantic representation is decisive: a domain-focused embedding strategy yields near-perfect campaign separation, substantially outperforming full-URL representations. On live, in-the-wild threat feeds, the domain-focused representation recovers all 944 discovered campaigns at an Adjusted Rand Index of 0.990 and a mean campaign recall of 1.000, at 0.10 ms per URL. Under identical clustering, the full-URL representation reaches an Adjusted Rand Index of only 0.510 and recovers fewer than half of the campaigns. We add a SHAP-based explainability layer that attributes discovery decisions to interpretable structural patterns, and we expose the whole pipeline as an operational system with a REST interface for single- and batch-URL analysis. The encoder we employ is a discriminative representation model rather than a generative one; what we contribute is the representation-and-modeling discipline this setting demands, and the resulting guidance transfers to threat intelligence in an era of generatively produced attack campaigns.