TabFM-Auto is introduced, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it, and ranks first overall among MLE agents.
Abstract
Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.
TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning, produces calibrated zero-shot predictions in a single forward pass without task-specific tuning.
Wei-Hao Kong, Erez Louidor Ilan, Shu-Xin Nie et al.· 24 citations· ⚡4
This review is, to the authors' knowledge, the first organised around the current generation of tabular foundation models, and taxonomises the architectures by pretraining regime, maps the capability space across five axes, isolates the language-model-on-tabular strand for prediction, feature engineering and generation...
S. Kurashkin, V. Tynchenko, Alexey S. Borodulin et al.· Machine Learning and Knowled...· 1 citation
Tabular foundation models achieve state-of-the-art performance on single-table tasks without any training. Recent work suggests that they are also well-suited for relational learning via deep feature synthesis (DFS), which flattens a relational schema into a single table by adding aggregates of the other tables'columns...
It is found that the optimal backend choice differs between column and row attention and varies across hardware as well as model specifics: While the FlashAttention implementations tailored for each GPU generation perform overall best, they are at times outperformed by CuDNN in the case of column attention at longer se...
Maximilian Schambach, Clemens Biehl, Sam Thelin· 0 citations
A recipe under which the latent term of a joint-embedding predictive architecture (JEPA) collapsed in earlier runs and took the encoder with it to a constant map is reported, under which the value head reads the encoder field rather than the predictor, and the target is an exponential moving average difference.
A controlled study across several versions of two major TFM families finds a consistent pattern: feature engineering gains are concentrated in earlier model generations and become negligible for the strongest models, suggesting that stronger TFMs depend less on explicitly engineered input representations.
Yi-Fan Wu, Pin-Jun Dong, Jia Tao et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026