From Nimitz to NetPila: The Evolution of Production-Scale Container Network
TL;DR
By embedding container addresses in the 128-bit IPv6 space, NetPila integrates the container network with the physical network, reducing packet-processing complexity and improving scalability.
Abstract
This paper describes two generations of production container networks deployed for over five years. Our first-generation network, Nimitz, used VxLAN for overlay-underlay mapping, providing flexibility and scalability from 2019-2022. However, with the rise of large-scale services (e.g., AI training and inference), Nimitz hit the C100K problem: beyond O(100K) containers, address mapping overhead and complex packet processing caused significant performance degradation. To overcome this, we built NetPila, a second-generation design that uses simple, practical IPv6 addressing to remove VxLAN encapsulation and overlay-underlay mapping tables. By embedding container addresses in the 128-bit IPv6 space, NetPila integrates the container network with the physical network, reducing packet-processing complexity and improving scalability. NetPila now supports millions of containers per tenant for production AI training and inference. We present the design, lessons, and deployment results of both generations.