Enzyme Demo: Incremental View Maintenance for Data Engineering
TL;DR
This demonstration presents Enzyme, the incremental view maintenance (IVM) engine powering Spark Declarative Pipelines, which builds on top of Apache Spark primitives and offers an end-to-end optimization layer for the selection of optimal refresh strategies corresponding to a collection of materialized views organized into a dataflow pipeline.
Abstract
This demonstration presents Enzyme, the incremental view maintenance (IVM) engine powering Spark Declarative Pipelines. While most industrial systems lack broad SQL operator coverage or require users to hand-tune refresh strategies, Enzyme provides an out-of-the-box solution for incremental ETL pipelines. Beyond classical delta-plan construction, it introduces techniques for incrementalizing non-deterministic operators such as temporal filters, a multi-versioned fingerprinter stable across system upgrades and multilingual MV definitions, and an adaptive cost model that leverages historical execution profiles to optimize refresh strategies across entire dependency DAGs. This leads to a reduced total cost of ownership (TCO) for users, who can focus on developing business logic instead. Enzyme builds on top of Apache Spark primitives and offers an end-to-end optimization layer for the selection of optimal refresh strategies corresponding to a collection of materialized views organized into a dataflow pipeline. The design is modular and can be easily extended to a variety of data sources and query processing engines.