VCF-RDFizer evaluation: benchmark harness and run records

Abstract

The evaluation of VCF-RDFizer reported in VCF-RDFizer: From a VCF File to Explicit, Verifiable, and Policy-aware Semantic Genomic Data: every number, table, and figure in the manuscript and its supplement can be traced to a run record in this archive and recomputed from it. Contents benchmarks/: the benchmark harness, experiments 00-19, including the linked workflow (benchmarks/use_case/acmg/) and the shared-input converter comparison (benchmarks/converters/). benchmarks/README.md describes each experiment and the reported result it produced; benchmarks/DESIGN.md gives each experiment's design as run, organized by the manuscript's three research questions. The drivers that ran the experiments on each benchmark host are archived with their run records. benchmark-results/: the run records of every reported run, per host: commands, tool commits, image digests, timings, resource samples, validation reports, query results, comparisons, and logs. Generated graphs and indexes are not included. benchmark-results/README.md maps every reported result to its records. scripts/figure_data.py and scripts/build_site_data.py: compute every plotted and quoted value from the run records alone; scripts/test_build_site_data.py checks them against the manuscript. plugin-tests/: tests of VCF-RDFizer's linking and policy plug-ins on real data; site/: the results site, whose every number is computed from the run records. Inputs: the download command for every input VCF (README.md), their SHA-256 checksums (benchmark-results/input-checksums.tsv and vcf-input-sizes.json), and the scripts that derive the evaluated slices. Software: the core benchmark (experiments 00-13) ran VCF-RDFizer v3.1.0 (image ecrum19/vcf-rdfizer@sha256:1904e96dde12ab2e2e70d8ee1267765c293ab100a8bd1b14d5b009b2bf8e34aa); the linked workflow, regional and large-graph retrieval, the converter comparison, the consumer WGS validation run, and the index-storage measurement used VCF-RDFizer v3.3.1 (image ecrum19/vcf-rdfizer@sha256:3ad71b1a54612142e3be43b24e7e4a38551949cb641f4421bcae9bf051102993). The two releases write identical graphs: their conversions of the 100,000-record HG005 slice have the same 17,098,746 triples (sorted N-Triples SHA-256 661578e7…94ed). Not included: runs whose results the manuscript does not report (pre-release, superseded, stalled and failed runs, including those the supplement cites as evidence of corrected defects), which are on the GitHub repository's legacy branch; the manuscript, its figures and figure scripts, and the authors' working notes, which stay in the GitHub repository (BioMedSem_2026/, tool-docs/; marked export-ignore in .gitattributes); the source VCFs, which remain available from their providers (Genome in a Bottle, the 1000 Genomes Project, HGSVC2, precisionFDA, and the Personal Genome Project); the generated RDF graphs and indexes, which the archived commands regenerate; and the Docker images, which are pulled by digest. Licences: code under the MIT License (LICENSE); data and documents under CC BY 4.0 (LICENSE-DATA), except the recorded MyVariant.info responses in benchmarks/use_case/acmg/myvariant-cache/, which remain under MyVariant.info's terms of use.