An analytical survey of algorithms for efficient metadata indexing and search in distributed file systems
TL;DR
Evaluated on a 12-node HPC cluster using genomics, climate, and synthetic datasets, TagIt++ demonstrates significant improvements, including a 55% reduction in search latency, 38% higher indexing throughput, and 96% metadata coverage.
Abstract
High-performance computing (HPC) environments generate large volumes of heterogeneous data, challenging traditional metadata management in distributed file systems (DFSs). Existing portable operating system interface (POSIX)-based metadata models offer limited support for semantic queries and content-based search, leading to reliance on external crawlers or centralized services that introduce latency and scalability issues. This paper presents TagIt++, an extension of the TagIt framework, which integrates metadata indexing directly within DFS volume servers. TagIt++ introduces automated semantic metadata enrichment, locality-aware federated indexing, and secure in-situ operator execution without modifying the underlying file system. Evaluated on a 12-node HPC cluster using genomics, climate, and synthetic datasets, TagIt++ demonstrates significant improvements, including a 55% reduction in search latency, 38% higher indexing throughput, and 96% metadata coverage. Query accuracy improves by 6.7% (F1-score), while storage overhead is reduced by 25%. Ablation results confirm the effectiveness of each component.