Unlocking an Archive of Scholarly Journal Articles with Interactive XQuery

Abstract

A BaseX interactive XQuery environment is used to access and query a content archive of more than 5.5 million scholarly journal articles stored in a variety of XML formats. The BASEX database records metadata for each document (dual metadata if needed for different use cases), supporting fast index-based retrieval and synchronization with the content repository. The database offers two complementary interactive user interfaces: web-based and Jupyter Notebook. The web-based interface provides everything from simple document retrieval (by identifier such as DOI or filename) through complex queries across the corpus, including the ability to extract specific portions of selected documents. Available web features include predefined reports, on-screen help, and status monitoring, while also enforcing security restrictions on user-provided XQuery. The Jupyter Notebook interface uses a custom Jupyter Kernel which parallels the functionality of the web interface and enables iterative workflows for data exploration that can provide text, code, and output integrated in a single document. I describe successful design decisions and lessons learned that may benefit others implementing similar XML database solutions.