Sep 2026· IEEE transactions on computers· Vol 75, pp. 3055-3069· 0 citations· 43 references
Abstract
Deep Neural Networks (DNNs) have become foundational to intelligent systems, yet deploying them efficiently under strict latency, resource, and privacy constraints remains challenging. While cloud-only inference suffers from transmission latency and privacy risks, and edge-only execution is limited by hardware capacity, cloud–edge collaborative inference offers a practical middle ground by combining the cloud’s compute strength with the edge’s proximity to data sources for low-latency, scalable, and privacy-aware inference. However, realizing this potential requires adaptive DNN partitioning that responds to dynamic workloads and network conditions, as well as fine-grained cross-tier resource orchestration to avoid bottlenecks and ensure system stability. To this end, we propose DDPG-DRPA-driven Deep Neural Network(D3NN), a novel and efficient framework for partitioned DNN deployment across cloud and edge resources. We formulate the pipeline partitioning of DNNs as a Markov Decision Process (MDP). A value function is trained using the Deep Deterministic Policy Gradient (DDPG) algorithm, and a Dynamic Resource Partitioning Agent (DRPA) allocates suitable cloud or edge resources to each DNN layer according to specific task types. As a result, D3NN adapts dynamically to both environmental conditions and task requirements. Under maximum task arrival rate scenarios, our approach reduces inference latency by 13.7% compared to pure cloud-based inference and by 33.5% compared to pure edge-based inference, demonstrating its practical effectiveness in resource-constrained cloud–edge systems.
As deep learning permeates latency-sensitive domains such as autonomous driving and smart surveillance, deploying neural networks (NNs) across the computing continuum (CC), from IoT devices to edge servers and cloud platforms, has become increasingly important. In such heterogeneous IoT-Edge-Cloud environments, distributed inference promises reduced latency, improved privacy, and better resource utilization. Yet, determining how to deploy NNs over heterogeneous IoT-Edge-Cloud nodes remains a difficult and largely manual process. This paper presents a principled and extensible framework for evaluating distributed inference of NNs in heterogeneous CC infrastructures. We introduce a formal model that unifies functional, pipelined, and data-parallel partitioning strategies within a single abstraction over heterogeneous CC topologies, enabling structured cross-strategy comparison. Building on this foundation, we implement a distributed inference orchestrator that supports flexible deployment of partitioned CNNs, and introduce PartiBench, a benchmarking tool that profiles segments and guides their placement. Our evaluation demonstrates how the framework exposes key performance trade-offs, offering actionable insights into latency, memory use, and communication overhead across IoT-Edge-Cloud nodes. These contributions enable empirical, cross-strategy comparison of distributed inference deployments and provide a basis for future automated placement methods in heterogeneous IoT-Edge-Cloud systems.
Nikolaos Papadakis, Alexandros Angourakis, K. Magoutis et al.· International Conference on...· 0 citations
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
Reza Farahani, Zoha Azimi, Mario Colosi et al.· 0 citations
The increasing deployment of artificial intelligence (AI) applications in healthcare, industrial Internet of Things (IIoT), intelligent transportation, and next-generation wireless systems has created a demand for inference architectures that simultaneously provide low latency, scalability, privacy, reliability, and efficient resource utilization. Conventional cloud-centric inference architectures provide substantial computational capacity but can introduce network latency, bandwidth consumption, privacy exposure, and dependence on centralized infrastructure. Edge computing addresses several of these limitations by relocating computation closer to data sources, while cloud environments remain important for computationally intensive and globally coordinated workloads. This research examines a scalable edge-to-cloud AI inference pipeline in which inference tasks are dynamically distributed across heterogeneous edge and cloud resources. The methodology synthesizes the provided literature on federated learning, edge resource allocation, dynamic scheduling, privacy preservation, machine learning for 6G, IIoT, and secure healthcare systems. A layered architectural model is developed around workload characterization, adaptive task placement, communication-aware scheduling, privacy protection, and resilient orchestration. The analysis indicates that scalability is not achieved merely by adding computational resources; rather, it depends on coordinated optimization of computation, communication, privacy, and scheduling. The proposed conceptual framework positions edge inference as the first computational layer, cloud inference as an elastic computational layer, and intelligent orchestration as the mechanism connecting the two. The resulting architecture provides a basis for resilient real-time AI systems while highlighting unresolved challenges involving heterogeneous hardware, dynamic workloads, privacy-utility trade-offs, and cross-layer optimization.
Dr. Khalid Al- Mansour· International Journal of Com...· 0 citations
Large-small model collaboration, as an emerging paradigm for efficient language model inference, plays a significant role in balancing inference quality and service efficiency in cloud-edge intelligence systems. By combining the strong generalization capability of cloud-side large language models (LLMs) with the low-latency inference capability of edge-side small language models (SLMs), this paradigm provides a practical way to serve heterogeneous language requests. However, existing collaborative inference methods still face a critical challenge: forwarding all requests to the cloud leads to high latency and unnecessary service cost, while relying only on edge-side SLMs may degrade prediction quality for difficult inputs. Therefore, we propose DABO, a calibration-aware binary offloading method for collaborative large–small model inference. Specifically, the edge-side SLM first performs lightweight probe inference and produces a prediction confidence distribution. DABO then calibrates the confidence through temperature scaling and converts it into a label-space-normalized difficulty score. Based on a model–task-specific operating threshold selected under an explicit accuracy constraint, reliable requests are answered locally by the edge-side SLM, whereas uncertain requests are selectively escalated to the cloud-side LLM. Extensive experiments demonstrate that DABO maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge, reducing expected calibration error by 31.08%, and decreasing mean latency by 84.24% compared with Cloud-only inference in a controlled workstation-based edge simulation.
Chen Zhu, Yi-Ming Su, Chenwenjie Mao et al.· IEEE Access· 0 citations
The rapid advancement of large language models (LLMs) has increased demand for scalable and cost-effective deployment, especially for mobile and edge devices. Cloud-hosted LLMs are powerful but expensive and difficult to scale due to vendor lock-in and high resource needs, resulting in high expenses and unstable performance under load. Recent efforts focus on deploying small language models (SLMs), distilled or pruned from LLMs, on resource-constrained edge devices to reduce costs and improve scalability. However, edge-based SLMs face limited knowledge coverage and notable accuracy gap compared to cloud-based LLMs. To address this, we present DEFRAG, a decentralized edge collaboration system for retrieval-augmented generation (RAG) that optimizes both retrieval and generation across heterogeneous edge devices. For retrieval, DEFRAG compresses and shares knowledge graphs, using hybrid retrieval to expand knowledge coverage. For generation, DEFRAG introduces an optimizer that adaptively selects SLMs and RAG parameters per query, balancing accuracy and cost. We implement DEFRAG on a heterogeneous edge testbed and evaluate it on benchmark QA datasets. We also test it under mobile route stress, non-uniform data placement, and a domain-specific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services. These findings demonstrate the potential of DEFRAG for democratized LLM services at the edge.
Jiaxing Li, Hengzhi Wang, Feng Wang et al.· IEEE Transactions on Mobile...· 0 citations
Traditionally, training and hyperparameter optimization of deep neural networks rely heavily on centralized cloud data centers. However, privacy concerns are driving a paradigm shift to move model fine-tuning directly to localized edge environments. Migrating Population-Based Training (PBT) to the edge presents severe challenges: unlike uniform cloud servers, edge networks exhibit extreme hardware heterogeneity. While Low-Rank Adaptation (LoRA) accelerates individual trials on weak devices, permanently maintaining adapters causes structural divergence during PBT weight inheritance. To cope with these edge-specific constraints, we propose Dynamic LoRA-PBT, an asynchronous hardware-software co-design. Systematically, it mitigates extreme evolutionary staleness via a capability-aware scheduler and a late-stage CPU dropout mechanism. Algorithmically, it introduces a Merge-and-Unload strategy, injecting LoRA for early exploration and explicitly merging it into the dense model before mutation. Preliminary evaluations on an 11-node CPU/GPU edge-server cluster demonstrate a 23.4 percent reduction in Time-to-Accuracy (TTA). As an ongoing work, we are currently integrating Transformer architectures and conducting rigorous statistical validations to solidify this framework.
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.