Profiling Neural Network Partitioning Strategies for Inference across the Computing Continuum
TL;DR
A formal model that unifies functional, pipelined, and data-parallel partitioning strategies within a single abstraction over heterogeneous CC topologies, enabling structured cross-strategy comparison and enabling empirical, cross-strategy comparison of distributed inference deployments is introduced.
Abstract
As deep learning permeates latency-sensitive domains such as autonomous driving and smart surveillance, deploying neural networks (NNs) across the computing continuum (CC), from IoT devices to edge servers and cloud platforms, has become increasingly important. In such heterogeneous IoT-Edge-Cloud environments, distributed inference promises reduced latency, improved privacy, and better resource utilization. Yet, determining how to deploy NNs over heterogeneous IoT-Edge-Cloud nodes remains a difficult and largely manual process. This paper presents a principled and extensible framework for evaluating distributed inference of NNs in heterogeneous CC infrastructures. We introduce a formal model that unifies functional, pipelined, and data-parallel partitioning strategies within a single abstraction over heterogeneous CC topologies, enabling structured cross-strategy comparison. Building on this foundation, we implement a distributed inference orchestrator that supports flexible deployment of partitioned CNNs, and introduce PartiBench, a benchmarking tool that profiles segments and guides their placement. Our evaluation demonstrates how the framework exposes key performance trade-offs, offering actionable insights into latency, memory use, and communication overhead across IoT-Edge-Cloud nodes. These contributions enable empirical, cross-strategy comparison of distributed inference deployments and provide a basis for future automated placement methods in heterogeneous IoT-Edge-Cloud systems.