Computation Is Fast, Use Threadlet!: Efficient Threading for μs-Scale Computing via OS/Hardware Co-Design

Abstract

The conventional threading model multiplexes software threads onto hardware cores. This model inherently suffers from the overhead of 1) context switching, 2) scheduling, and 3) system event notifications (e.g., I/O interrupts). As computing enters μs-scale, such overheads become the key bottleneck of a wide range of domains: networking, concurrency control, and microservice scheduling, etc. Existing software solutions, such as cooperative scheduling and polling, sacrifice correctness and/or CPU efficiency, while hardware offers only point solutions with suboptimal performance. To eliminate the threading overheads without introducing difficult tradeoffs, our key insight is that hardware should directly expose schedulable, basic units of execution (i.e., software threads) instead of computational resources (i.e., cores). We propose threadlet, a new hardware abstraction that is lighter than a software thread by removing uncritical state (e.g., floating-point registers). A core caches a few active threadlets in its register file, while spilling the rest to the memory hierarchy. Thus, context switching incurs minimal overhead by simply resuming execution with the next threadlet's state. Moreover, threadlet comes with hardware support that can express diverse OS scheduling policies, enabling fast and flexible scheduling. Finally, as a hardware abstraction, threadlets directly interface with microarchitectural mechanisms (e.g., cache coherence) for efficient event notifications. We build a supporting architecture on an FPGA and a research OS for threadlet, outperforming prior works by 3× to 47× on real-world applications. Our artifact is publicly available at https://github.com/TELOS-syslab/Threadlet.