Simon Dumas Primbault

Auditing Declared AI Generative Crawler Traffic on a Large-Scale Multilingual Digital Library

Generative AI has brought a new wave of automated visitors to digital libraries, and libraries now try to allow, slow or block particular crawlers. The tools they use, such as robots.txt rules and block lists, act on the name a crawler gives itself. That name is a text string any client can send, so it cannot prove ide...