Efficient Piecewise-Linear Embeddings for Deep Tabular Regression by Guided Breakpoint Allocation

Abstract

Tabular prediction is central to scientific and industrial decision-making, and many high-impact use cases are regression tasks that predict continuous targets. Deep tabular networks for regression tasks rely on effectively modeling irregular and non-smooth relationships between numerical features and continuous targets. To this end, numerous numerical embeddings have been proposed. However, their performance is often highly sensitive to parameter choices, such as the choice of frequencies for Fourier features and the placement of breakpoints for piecewise-linear embeddings. Moreover, the irregular and non-smooth characteristics of tabular data make it difficult for gradient-based end-to-end training to discover effective parameter settings for these embeddings. In this work, we demonstrate the underexplored potential of piecewise-linear embeddings, showing that finding better breakpoints alone can yield substantial gains. We introduce GBDT-Guided Piecewise-Linear (GGPL) embeddings, which leverage GBDTs to provide a strong data-driven prior for breakpoint placement. To further fine-tune these breakpoints via gradient descent, GGPL reparameterizes them for numerical stability and regularizes training via stochastic breakpoint deactivation. Across 28 regression datasets, integrating GGPL with diverse state-of-the-art deep tabular models yields consistent and significant improvements. GGPL is applied only at training time, introducing no inference overhead. Furthermore, in controlled MLP experiments, it uses only 0.4× as many breakpoints on average as prior piecewise-linear embeddings, while achieving higher accuracy. Combined with its negligible overhead, these results establish GGPL as an effective numerical embedding for deep tabular regression.