SynapticOS: An Inference-First Runtime Architecture for Neural Processing Units on Resource-Constrained Microcontrollers

Dimitrios Kafetzis

Jul 2026 · arXiv.org · Vol abs/2607.12606 · 2 citations · 19 references

Computer Science Engineering

TL;DR

This work presents the Phase 1 foundation of SynapticOS, an open-source runtime built on Zephyr that treats inference as a first-class workload and evaluates on the NXP FRDM-MCXN947 and the qemu_cortex_m3 emulator.

Abstract

Microcontrollers with on-die neural processing units (NPUs) have become mainstream, but the system software hosting them has not: production combinations of Zephyr or FreeRTOS with TensorFlow Lite Micro treat AI inference as an application-layer library, leaving memory fragmentation, accelerator-state hygiene, and model-lifecycle guards as recurring application-developer concerns. We present the Phase 1 foundation of SynapticOS, an open-source runtime built on Zephyr that treats inference as a first-class workload. It contributes four cooperating subsystems: (1) a tensor-aware bump allocator with 16-byte DMA-aligned persistent and ephemeral lifetimes sharing a single arena, achieving constant-time allocation (~154 cycles per call, ~78,000 allocations per second at 150 MHz, invariant across tensor sizes) with zero fragmentation by construction; (2) a four-state hardware abstraction layer for the NPU and DSP, implemented by a deterministic software stub (for CI under QEMU) and a Neutron-flavoured backend (for the NXP MCXN947); (3) a three-state model lifecycle registry with duplicate-name detection, idempotent load/unload, and hot-swap guards; and (4) a four-mark cycle-accurate profiler. We evaluate on the NXP FRDM-MCXN947 (dual Cortex-M33 at 150 MHz) and the qemu_cortex_m3 emulator. Build footprints are 67 KB flash / 184 KB SRAM on FRDM (shell, 128 KB arena) and 24 KB flash / 28 KB SRAM on QEMU (no shell, 8 KB arena). End-to-end inference brackets through the deterministic stub kernel measure 1,038 us on FRDM and 781 us on QEMU for a 16x16x3 INT8 input; these are baseline overhead numbers, not Neutron silicon measurements, which arrive with the real SDK invoke path in Phase 2. A 61-test suite across 10 ZTEST suites passes 100% in 6.6 s on the CI emulator path. SynapticOS is released under Apache 2.0 at https://github.com/Dimitrios-Kafetzis/SynapticOS

DOI arXiv Semantic Scholar

View source

Similar papers

Open access Oct 2026

QUINN: Quantized Inference and Dataflow Characterization for Neural Networks on RISC-V Near-Memory Architectures

Machine learning inference is moving to the edge due to latency, energy, and privacy constraints. The growing complexity of edge workloads places increasing pressure on memory hierarchies, making data movement a major limitation for both performance and energy efficiency. Fixed-function accelerators inherit the memory...

Vincenzo Petrolo, Flavia Guella, Guido Masera et al. · Computers · 0 citations

Preprint Aug 2026

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

This report argues that the most effective response to single-token autoregressive decode on CPUs is to co-design the model architecture and the inference runtime together, and presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency grap...

Tom Poperszky · 0 citations

Aug 2026

Hardware-Aware Neural Network Deployment on Multi-Core in-Memory Computing Systems: A Compiler Perspective

Conventional compute-in-memory (CIM) deployment flows usually assume a fixed crossbar geometry, although DNN layers often have different channel counts and matrix shapes. The resulting shape mismatch leaves part of the array capacity unused and can increase the number of split-and-transfer operations during inference....

Kaiwen Deng, Sifan Sun, Hanjie Liu et al. · IEEE Non-Volatile Memory Sys... · 0 citations

Conference Sep 2026

QXL - A Scalable Hardware Framework for Tabular Q-Learning Inference on RISC-V

Tabular Q-learning is a foundational reinforcement learning (RL) algorithm used in embedded decision-making, yet its inference phase suffers from latency-sensitive memory accesses, creating severe bottlenecks in real-time control systems. To address this, we present QXL, a hardware-software codesigned inference acceler...

C. R, Pruthvi Parate, P. K. et al. · IEEE International Conferenc... · 0 citations

Preprint Sep 2026

The World Model Hardware Accelerator

WMHA is exploited in WMHA, a latency-first diffusion-transformer inference accelerator: a very-long-instruction-word sequencer issues four engines from one instruction word, a weight-stationary 16x16 dual-dot array streams FP8 and BF16 contractions, and a single-pass online-softmax attention pipeline keeps keys and val...

Shashank Chaurasia · 0 citations

Review Aug 2026

AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall

This review surveys the full landscape of AI hardware accelerators for LLMs, including general-purpose GPUs, custom ASICs, reconfigurable FPGAs, processing-in-memory and near-memory architectures, and emerging neuromorphic and photonic approaches across cloud and edge deployment.

Siddharth Patel, Rohit Singh · 0 citations