TabNSM provides an effective and scalable approach to deep tabular regression, and demonstrates that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.
Abstract
Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.
Tabular learning is still dominated by gradient-boosted decision trees (GBDTs), while recent deep learning approaches have become increasingly competitive. However, applying deep tabular models to large-scale datasets remains challenging, as large sample sizes, high feature dimensionality, or many target classes can introduce substantial computational cost. We propose TabLoRA, a parameter-efficient trainable neural ensemble for large-scale tabular learning. Instead of using fully independent ensemble backbones, TabLoRA shares a common backbone across predictors and introduces predictor-specific low-rank adaptations, enabling ensemble-style prediction without full parameter duplication. Across benchmarks, TabLoRA achieves a favorable balance between predictive performance and practical efficiency compared with GBDT methods and recent deep learning baselines under the same resource constraints. Memory analysis and ablation studies further show that the proposed design improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles.
Tabular prediction is central to scientific and industrial decision-making, and many high-impact use cases are regression tasks that predict continuous targets. Deep tabular networks for regression tasks rely on effectively modeling irregular and non-smooth relationships between numerical features and continuous targets. To this end, numerous numerical embeddings have been proposed. However, their performance is often highly sensitive to parameter choices, such as the choice of frequencies for Fourier features and the placement of breakpoints for piecewise-linear embeddings. Moreover, the irregular and non-smooth characteristics of tabular data make it difficult for gradient-based end-to-end training to discover effective parameter settings for these embeddings. In this work, we demonstrate the underexplored potential of piecewise-linear embeddings, showing that finding better breakpoints alone can yield substantial gains. We introduce GBDT-Guided Piecewise-Linear (GGPL) embeddings, which leverage GBDTs to provide a strong data-driven prior for breakpoint placement. To further fine-tune these breakpoints via gradient descent, GGPL reparameterizes them for numerical stability and regularizes training via stochastic breakpoint deactivation. Across 28 regression datasets, integrating GGPL with diverse state-of-the-art deep tabular models yields consistent and significant improvements. GGPL is applied only at training time, introducing no inference overhead. Furthermore, in controlled MLP experiments, it uses only 0.4× as many breakpoints on average as prior piecewise-linear embeddings, while achieving higher accuracy. Combined with its negligible overhead, these results establish GGPL as an effective numerical embedding for deep tabular regression.
Min-Kook Suh, Moonjung Eo, Kyungeun Lee et al.· Proceedings of the 32nd ACM...· 0 citations
Pretrained tabular foundation models have demonstrated strong predictive capability; however, their application to large-scale datasets remains constrained by the limited inference context. This paper introduces Balanced Adaptive Prototype Selection (BAPS), a framework for constructing compact, information-preserving contexts for scalable TabPFN inference. Without modifying or retraining the pretrained model, BAPS jointly preserves representative structure, informative decision boundaries, local density, class balance, and feature-space diversity. Experiments on the million-row HIGGS and SUSY datasets show that 512 prototypes retain strong predictive performance and reliable calibration, corresponding to an approximately 1,953-fold context compression. All experiments were conducted on an Intel Core i7 CPU with 16 GB RAM and no GPU acceleration. These findings establish effective context construction as a practical mechanism for extending pretrained tabular foundation models to million-scale datasets.
Mahboobe Jadid, Melika Rezaye Garkani, A. Mousavi· 0 citations
Results support IAIML as a compact, interaction-aware framework appropriate for settings where bounded explanation size and controlled treatment of feature interactions are design requirements.
EXAONE Tabular is a compact tabular foundation model family for classification and regression via in-context learning, producing predictions without dataset-specific gradient updates. Pretrained exclusively on a synthetic structural-causal-model (SCM) prior, its central contribution is an architecture-centered redesign of tabular in-context learning. Rather than compressing features into a fixed row embedding before a separate row-level learner, EXAONE Tabular interleaves feature-axis attention within each item with support-conditioned item-axis attention within each feature at every Transformer layer, mediated by item-summary and feature-summary tokens. Across four public benchmarks, EXAONE Tabular combines strong predictive performance with high efficiency. On TabArena, its 20.81M-parameter classification model ranks first overall, surpassing tuned ensembles and 4-hour AutoML pipelines, while regression reaches the performance regime of the 1.64B-parameter TabFM at roughly 1/11 the inference cost. On BCCO and TALENT, EXAONE Tabular ranks second in classification and first in regression. On ScoringBench, it achieves the best mean rank for both point-estimation and predictive-distribution quality, leading the $R^2$, RMSE, and CRPS evaluations. Together, these results establish EXAONE Tabular as a state-of-the-art compact tabular foundation model family, combining strong predictive performance across classification, point regression, and probabilistic regression with an efficient model design.
Moonjung Eo, Min-Kook Suh, Hye-Seung Cho et al.· 0 citations
An Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained LLMs into depth-sparse models without full retraining and consistently yields smaller accuracy degradation than strong baselines including MoD, D-LLM, and DLO.
Yi-Da Wu, Xiang Wang, Kejie Zhao et al.· International Conference on...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
MIT News · Artificial Intelligence· news.mit.eduAug 24, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
