Oasis is a data-processing SmartNIC that offloads Parquet decoding into the network datapath as a custom hardware accelerator, and shows that Oasis hides the cost of Parquet decoding behind the network datapath with minimal overhead, overlapping the scan with the remainder of the query execution.
Abstract
Cloud-native database systems disaggregate compute and storage resources to improve cost efficiency over traditional monolithic architectures through elasticity and resource pooling. Studies of production data warehouse workloads show that scans (including round trips to storage) account for roughly half of total query runtime. Data lakes and lakehouses amplify this bottleneck through per-query decoding of storage-optimized, compressed file formats such as Parquet. As storage and network bandwidth continue to outpace CPU cost-performance, the CPU cycles spent on decoding increasingly undermine the cloud's cost-efficiency promise. This has led to a wave of specialization across the stack with custom hardware at cloud-vendor scale at the extreme end. We build on this trend and present Oasis, a data-processing SmartNIC that offloads Parquet decoding into the network datapath as a custom hardware accelerator. Oasis features a hardware decoder architecture, software abstraction layer, and end-to-end integration with DuckDB. Our evaluation shows that Oasis hides the cost of Parquet decoding behind the network datapath with minimal overhead, overlapping the scan with the remainder of the query execution. In the best case, this almost doubles DuckDB query throughput.
It is found that hardware offload is not a universal replacement for CPU compression, and compression should be scheduled dynamically: route blocks by codec, operation, size, and accelerator load; cap per-device sub-mission concurrency; and fall back to software when offload is unsupported or saturated.
Yi Jiang, Antonio Boffa, Hamish Nicholson et al.· 0 citations
The convergence of unified, cloud-native Lakehouse platforms such as Microsoft Fabric, and the widespread deployment of AI-optimized hardware in datacenters is driving the rise of GPU-based analytical engines. However, building a GPU query engine that is both competitive with hand-tuned implementations and portable across vendors (NVIDIA, AMD, custom silicon) remains an open challenge, as existing systems either lock into a single vendor's toolchain or sacrifice performance for generality. In this paper, we present TQP++, an ML-compiler-native analytical query processor that repurposes ML compiler infrastructure to close this gap, achieving high performance and hardware portability from a single codebase. TQP++ integrates the Antares compilation framework with tiered GPU resource scheduling for SQL operators, a map-reduce-oriented fusion schema that eliminates intermediate materializations, and a multi-gated execution graph that adapts operator algorithms to runtime data characteristics. On TPC-H SF 100, TQP++ executes all 22 queries in 1.1 second on an A100 (7× faster than HeavyDB, 15× faster than CPU baselines) and under 0.7 seconds on H100 and MI300, while targeting 9 devices across 3 vendors—including Xbox—without code changes. To our knowledge, this is the most hardware-diverse GPU query processor reported to date, while achieving sub-second TPC-H SF100 on a single GPU.
Wei Cui, Peng Cheng, Carlo Curino et al.· Proceedings of the VLDB Endo...· 0 citations
Apache Parquet has significantly transformed big data processing with its efficient columnar storage capabilities, incorporating techniques such as the Partition Attributes Across (PAX) layout and dictionary-based encoding to enhance data compression and query performance. Despite these advantages, Apache Parquet's limited indexing capabilities can lead to inefficiencies when users need to fetch only a subset of the data. This paper proposes an innovative extension to Apache Parquet by introducing a new encoding format where indexes form the foundational physical representation of a column. Specifically, we introduce
overlay bitmaps
, a data structure that organizes column chunks into sequences of bitmap parts, each representing distinct column values that are further compressed using run-length encoding. Our implementation within the Apache Parquet C++ libraries and integration into the SAP IQ relational engine demonstrates the dual use of overlay bitmaps as both a storage and indexing mechanism, significantly improving predicate evaluation speed and compression efficiency. Experimental evaluations highlight that overlay bitmap encoding surpasses traditional dictionary encoding in compression and reduces load execution times by up to two orders of magnitude by enabling efficient, row-level, predicate pushdown. This capability allows complex queries, which involve numerous joins and predicates, to be executed up to 10x faster.
Günes Aluç, Daniel J. Farrar, Manoj Kumar Sharma et al.· Proceedings of the VLDB Endo...· 0 citations
Modern cloud applications and AI-driven workloads demand elastic compute, instant environment isolation, and open data interoperability. However, production databases remain largely based on monolithic architectures designed in the 1980s, tightly coupling storage and compute and creating operational fragility, cost inefficiencies, and vendor lock-in. This paper presents Lakebase, a new open database architecture that decouples transactional compute from storage while placing database data directly in low-cost cloud object stores in open format. Unlike second-generation cloud databases, such as Amazon Aurora, Microsoft Socrates and Google AlloyDB, which separate storage internally but retain proprietary formats and single-engine access, Lakebase exposes storage as an open, durable lake layer while running elastic, serverless Post-greSQL transactional engines independently on top. We describe the system design principles behind Lakebase, including open-format object storage persistence, compute elasticity down to zero, instant branching and cloning via copy-on-write semantics, and unified transactional-analytical access over shared lake storage, as deployed in the Databricks Lakebase service. We analyze how this architecture reduces operational complexity, enables Git-like database workflows, improves failure recovery semantics, and mitigates structural vendor lock-in. We further discuss production deployment characteristics, performance trade-offs, durability guarantees, and support for multicloud high availability. We evaluate Lakebase on standard OLTP and analytical benchmarks, alongside production telemetry on branching and elasticity. Lakebase represents a third generation of database architecture: transactional systems rebuilt natively for cloud object storage, elastic compute, and AI-era development workflows.
Jasraj Dange, Andrei Drâguş, Ali Ghodsi et al.· Proceedings of the VLDB Endo...· 0 citations
Hierarchical Data Storage Systems (HSSs) provide a cost-effective architecture that balances capacity and performance through internal data migration. Prior work has primarily focused on optimizing individual migration tasks, either within or across HSS tiers, or on exploiting device bandwidth to improve overall throughput. These approaches treat migration tasks in isolation, and the performance implications of executing heterogeneous migration tasks concurrently remain largely unexplored despite their prevalence in real-world HSS deployments. The growing adoption of Direct Data Access (DDA) architectures, in which accelerators access storage without CPU mediation, further amplifies this problem by removing a natural bandwidth arbiter from the I/O path.
This paper presents an in-depth analysis of data migration behavior in commercial HSSs, uncovering substantial performance variability when multiple migration tasks execute concurrently. To mitigate this issue, we propose PASCAL, a system-level bandwidth orchestration framework that improves performance robustness in production-grade HSSs. Inspired by hydraulic systems, PASCAL adapts pressure/backpressure-style coordination to the multi-task migration setting: it treats each tier as a pressurized vessel and uses pressure gradients to allocate bandwidth across cache flush, tiering, garbage collection, and DDA flows. We evaluate PASCAL on a commercial OceanStor HSS across three hardware configurations and eight workloads spanning database, AI training, AI inference, and production traces. PASCAL achieves up to 20% higher throughput, 67% lower tail latency, and 79% reduced throughput jitter compared to local state-of-the-art controllers, while also stabilizing the performance jitter introduced by DDA architectures.
Ji Zhang, Li Liu, André Brinkmann et al.· Proceedings of the VLDB Endo...· 0 citations
WADO is presented, a distributed write-once-read-many (WORM) object-store runtime for data-centric workloads that closes this gap through three coordinated mechanisms: scope-centric processing, explicit pipeline decomposition, and interference-aware explicit parallelism.
Karim Youssef, Hariharan Devarajan, Nikoli Dryden et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
