AdaKerNet is introduced, a novel learnable task-adaptive neural kernel decoder that operates solely on its (frozen) rich representations obtained from the diverse available modalities and established as an effective approach for prediction from frozen multimodal representations in the scarce label regime.
Abstract
Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an important class of large foundation models, remain challenging to adapt to various downstream tasks. Adaptation typically relies either on MLLM parameter fine-tuning or on training neural-based decoders. Both approaches struggle under limited supervision, while fine-tuning additionally requires access to model parameters, which is often unavailable for closed-source models. We introduce AdaKerNet, a novel learnable task-adaptive neural kernel decoder. AdaKerNet is fully agnostic to the parameters of the underlying MLLM and operates solely on its (frozen) rich representations obtained from the diverse available modalities. AdaKerNet relies on (i) a set of learnable, Lipschitz-controlled multimodal features derived from these MLLM representations; (ii) a reference kernel that provides a soft structural prior on those features; and (iii) a lightweight nonlinear neural predictor that adaptively deforms that structure. Learning the kernel representation and the neural predictor jointly within a unified optimization framework allows AdaKerNet to capture features and geometric relationships relevant to the downstream task. Numerical tests across four MLLMs: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2, and multimodal inputs spanning text, audio, images, and tabular measurements demonstrate significant and consistent improvements over direct MLP, attention-, autoencoder- and kernel-based decoders, across a range of scarce-label budgets, with average error reduction of up to 41% across baselines. These results establish AdaKerNet as an effective approach for prediction from frozen multimodal representations in the scarce label regime. Additional structural ablations highlight the complementary contributions of AdaKerNet's components.
Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.
Dezheng Han, Anlan Zhang, Zhiwu Zhu et al.· 0 citations
This paper tackles visible challenges in deep ensemble learning, where deep neural networks serve as ensemble members: training and storage burdens, and robustness of cautious (set-valued) predictions targeting multiple utilities, which may involve reward-sensitivity. To mitigate the training and storage burdens, we pr...
K. Tran, D. Nguyen, Vu-Linh Nguyen et al.· 0 citations
This work proposes Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data, and presents a gate based on a Gaussian Mixture Model, whose likelihood decays rapidly away from its training data, giving it a natural ten...
Assaf Ben-Kish, Akarsh Kumar, James R. Glass et al.· 0 citations
Bilateral Regularized Kernel Adaptation (BiReg), a closed-form, training-free framework that formulates few-shot VLM adaptation as multi-output kernel ridge regression, is proposed that achieves superior accuracy and robustness while retaining the efficiency of a closed-form, training-free solution.
ECHO-$k$ is introduced, a task-agnostic and self-supervised learning principle for modality acquisition that uses a deep model's internal pretrained representations as proxy targets that summarize cross-modal information and consistently improves budgeted downstream performance across diverse foundation-model backends.
Eeshaan Jain, L. Bleistein, B. Deplancke et al.· 0 citations
Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enab...
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
Exploring how generative AI could make machine vision more accessible to businesses. The post GenEye in a Box: Making Machine Vision Something You Can Just Ask For appeared first on GPT-Lab.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.