Fast and Accurate Classification with Parallel IDK Classifier Cascades

Abstract

Object recognition in edge systems requires a balance between prediction accuracy and strict decision time. Existing approaches like model compression, pruning, quantization, and early exit mechanisms can reduce computation but often require architectural modification of networks, retraining, or specialized deployment support, and most inference pipelines rely on sequential task execution, limiting their ability to exploit the task parallelism capabilities of modern GPUs. The IDK (”I Don't Know”) cascade addresses accuracy-decision time trade-offs by routing each input through a sequence of classifiers, enabling simple inputs to be resolved quickly while reserving computationintensive models for difficult cases. In this paper, we present a parallel IDK Cascade framework that extends the sequential cascade to parallel multistream GPU execution, increasing kernel execution overlap and lowering average decision time without changing model architectures or GPU runtime behavior. Experiments on ImageNet show that the proposed design reduces average inference time significantly while maintaining predictive accuracy.