GACER-M: Granularity-Aware and Memory-Centric Coordination for Multi-Tenant Deep Learning

Abstract

Multi-tenant computing, which executes multiple neural networks on a single GPU, is essential for maximizing hardware utilization in both cloud and edge environments. However, effective co-location faces a fundamental trade-off between utilization and safety. On one hand, the diverse computing patterns of concurrent models lead to resource fragmentation, requiring fine-grained scheduling to improve throughput. On the other hand, modern workloads (especially Generative AI) introduce strict memory constraints: their dynamic memory growth requires rigid reservation to prevent Out-Of-Memory (OOM) crashes, often leaving valuable resources idle but inaccessible to other tenants. Existing solutions struggle to balance these two sides: static partitioning ensures safety but limits utilization, while aggressive fine-grained scheduling boosts speed but risks system instability. To address this, we propose <bold><italic>GACER-M</italic></bold>, a unified resource coordination framework that harmonizes fine-grained computing with strict memory governance. Unlike previous approaches that sacrifice speed for safety, <italic>GACER-M</italic> achieves the best of both worlds via a tiered strategy: (1) A granularity-aware spatial mechanism that minimizes computational residue via operator splitting, strictly regulated by a cost-benefit guardrail to prevent memory bloat; (2) A memory-driven temporal coordination technique that segments dynamic workloads to safely inject concurrent tasks into resource gaps. <italic>GACER-M</italic> consistently achieves <bold>1.35</bold><inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives><mml:math><mml:mo>×</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq1-3701443.gif"/></alternatives></inline-formula><inline-formula><tex-math notation="LaTeX">$\sim$</tex-math><alternatives><mml:math><mml:mo>∼</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq2-3701443.gif"/></alternatives></inline-formula><bold>1.67</bold><inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives><mml:math><mml:mo>×</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq3-3701443.gif"/></alternatives></inline-formula> <bold>speedup</bold> in compute-bound scenarios compared to state-of-the-art works; furthermore, in memory-critical scenarios, it delivers <bold>1.33</bold><inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives><mml:math><mml:mo>×</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq4-3701443.gif"/></alternatives></inline-formula> <bold>higher effective capacity</bold> and <bold>1.4</bold><inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives><mml:math><mml:mo>×</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq5-3701443.gif"/></alternatives></inline-formula> <bold>aggregate throughput</bold> while ensuring strict memory safety.