Capacity-Aware Gradient-Boosted CNNs and DNNs for Efficient Image Classification
TL;DR
A capacity-aware GB-CNN (CW-GB-CNN) built on a closed-form capacity model of the additive dense head, Phead(W, M), which gives the head's parameter count from the layer width W and the number of boosting rounds M, and which achieved the best mean accuracy and mean rank on CIFAR-10.
Abstract
Gradient-boosted convolutional neural networks (GB-CNNs) grow a classifier by sequentially adding and freezing dense layers fitted to pseudo-residuals, yet the parameter cost of this additive head is seldom examined. This article proposes a capacity-aware GB-CNN (CW-GB-CNN) built on a closed-form capacity model of the additive dense head, Phead(W, M), which gives the head's parameter count from the layer width W and the number of boosting rounds M. The width of the boosted dense layers is selected by weighing the modeled parameter cost against the accuracy it returns, rather than being fixed a priori. CW-GB-CNN is compared with jointly trained networks, a gradient-boosted deep network, and classical tree ensembles on five public image datasets using rank-based statistical tests. Boosting reduced the test error of the jointly trained CNN by 49.7% on average, and CW-GB-CNN achieved the best mean accuracy (95.66%) and mean rank (1.6). On CIFAR-10, its 20-neuron head matched a conventional 128-neuron head (87.77% versus 87.86%) with 6.66 times fewer head parameters and 1.70 times fewer total parameters.