MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

Sana Tonekaboni Viktoria Schuster Caroline Uhler

Jul 2026 · arXiv.org · Vol abs/2607.16789 · 1 citation · 57 references

Computer Science

TL;DR

This work introduces MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models that produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

Abstract

Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

Structured Latent Modeling for Supervised Multimodal Information Decomposition

This work introduces a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model, offering a practical latent-variable lens for characterizing continuous multimodal interactions.

Wan-Ting Huang, Sanvesh Srivastava, Wei-Ran Wang · 0 citations

Open access Sep 2026

Cross-Modal Representation Learning for Integrating Heterogeneous Data in AI Systems

A cross-modal representation learning framework that aligns heterogeneous modalities within a shared latent representation space and exhibits strong robustness under missing modality conditions, with significantly lower performance degradation compared to baseline approaches is proposed.

I. Ibrahim, Marwan Alshar'e, I. Sanjaya et al. · Journal of Data Science · 0 citations

Preprint Aug 2026

Adaptive Hierarchical Representation Alliance for Multimodal Learning

Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework, is proposed, which consistently improves over strong baselines and remains robust under noisy and missing-modality settings.

Chun-Lei Meng, Peng-Bin Feng, Jacqueline J. Pang et al. · 2 citations

Book Open access Aug 2026

Retrv-MoE: Scaling Unified Multimodal Retrieval with Sparse Mixture-of-Experts

This work proposes Retrv-MoE, a unified retrieval architecture built upon sparse Mixture-of-Experts (MoE), and theoretically and empirically demonstrates that this conditional computation mechanism provides a structural remedy to optimization interference by decoupling the learning trajectories of conflicting tasks and...

Tongxu Lin, Jiayin Xiao · Proceedings of the 32nd ACM... · 0 citations

Conference Aug 2026

Adaptive Cross-Modal Fusion With Instance-Level Gating for Vision-Language Understanding

Multimodal deep learning integrates heterogeneous data sources such as images and text to enable machines to understand complex real-world contexts. Although recent vision-language models have achieved significant progress, most existing approaches rely on rigid fusion strategies that combine modalities either at early...

Unnati A. Patel, Sanskruti Patel, J. Nanavati et al. · International Conference on... · 0 citations

#machine learning Preprint Sep 2026

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes int...

Yavuz Yarici, Ghassan AlRegib · 0 citations