Virtualizing Recursion: Just-In-Time Graph Analytics in a Hyperscale Relational Warehouse

Abstract

Modern hyperscale data warehouses (MPP) face increasing pressure to support complex, non-linear query paradigms. While the industry is moving toward standardized graph query languages like ISO GQL and SQL:2023 (SQL/PGQ), supporting these on shared-nothing architectures traditionally requires specialized, heavyweight physical operators (e.g., recursive union). This complicates the scheduler and often forces a trade-off between latency and scale. Furthermore, in high-availability environments where metadata operations require distributed consensus, the latency of creating ephemeral tables for recursion becomes prohibitive. This paper presents Ocient's solution: decoupling the definition of recursion from the execution via Dataflows: a procedural superset of SQL. This architecture enables support for recursive SQL and is well-positioned as a transpilation target for emerging GQL and SQL:2023 workloads. We describe the implementation of a table virtualization runtime that inspects intermediate row counts to automatically switch between in-memory CTE injection (for low latency) and NVMe-backed storage (for massive scale). The system transparently transitions storage-backed tables back into virtual memory objects as they shrink or vice versa on growing tables. This unified substrate allows Ocient to execute hyperscale graph algorithms (such as Weakly Connected Components on graphs exceeding 100 billion edges) directly within the relational warehouse.