The Proactive Pipeline: Liquid Production Metadata for CG Pipelines
TL;DR
This work presents two complementary approaches: schema-flexible production metadata using JSONB within a validated relational database, and projection of relational state into a live dependency graph via Change Data Capture (CDC), which enable near–real-time propagation, resilience to partial failure, and scalability within a Proactive Pipeline architecture.
Abstract
As CG productions scale, teams depend on timely metadata to understand how work propagates across assets, shots, and files. Production data evolves continuously—new fields emerge mid-show, workflows change, and dependencies deepen—yet traditional relational systems struggle with costly schema changes and opaque transitive reasoning. This work addresses these challenges at the data-model level rather than through tools or interfaces. We present two complementary approaches: schema-flexible production metadata using JSONB within a validated relational database, and projection of relational state into a live dependency graph via Change Data Capture (CDC). Together, these enable near–real-time propagation, resilience to partial failure, and scalability within a Proactive Pipeline architecture. JSONB enables schema evolution without migrations or consumer breakage while supporting metadata exchange. In parallel, CDC streams relational state into a graph model, where dependency queries such as downstream impact and version reachability become natural traversals, enabling responsive, dependency-aware workflows.