Auditing Declared AI Generative Crawler Traffic on a Large-Scale Multilingual Digital Library

Abstract

Generative AI has brought a new wave of automated visitors to digital libraries, and libraries now try to allow, slow or block particular crawlers. The tools they use, such as robots.txt rules and block lists, act on the name a crawler gives itself. That name is a text string any client can send, so it cannot prove identity on its own. We ask how far a library can check a declared crawler identity against independent evidence: the IP ranges that operators publish. We apply this to the requests that OpenEdition, a large multilingual humanities and social sciences platform, identified as declared-bot traffic. Three lessons emerge. Among requests claiming a generative-AI identity, a minority is consistent with published operator infrastructure. Most come from families that publish no range at all, so they cannot be checked. And a mismatch is not necessarily spoofing: it can reflect one address posing as many bots, or an operator’s own list lagging behind its infrastructure. A library should therefore read a declared identity against the infrastructure documented at the time of the request. We release our pipeline and dated registry so that other libraries can run the same audit, and future studies can compare crawler traffic on a common footing.