The Evolution of Storage in Apache AsterixDB
TL;DR
This paper describes the evolution and current state of the storage subsystem of Apache AsterixDB, including the addition of column-oriented JSON storage, compute-storage separation with object store backing, and resource management policies targeting multi-user workloads over databases with many collections and indexes.
Abstract
This paper describes the evolution and current state of the storage subsystem of Apache AsterixDB. An initial shared-nothing version of AsterixDB, with an LSM-based storage system, was open-sourced in 2013 after four years of R&D; its storage system was described in a 2014 VLDB paper. In the dozen years since then, much has changed in the computing environment, including the move from magnetic storage to SSDs, the migration of database services to the cloud, and the availability and popularity of object storage. In this paper, we describe how AsterixDB storage has evolved in response, including the addition of column-oriented JSON storage, compute-storage separation with object store backing, and resource management policies targeting multi-user workloads over databases with many collections and indexes. Apache AsterixDB's storage system is also the storage system for the Capella Analytics and Enterprise Analytics product offerings from Couchbase, Inc.