2025· Neural Information Processing Systems· 6 citations· 58 references
Computer Science
TL;DR
A novel framework that performs selective cross-modal alignment through a learnable masking mechanism, enabling the model to isolate and align only the shared latent components relevant to both modalities is proposed.
Abstract
Text-to-Audio-Video (T2AV) generation aims to produce temporally and semantically aligned visual and auditory content from natural language descriptions. While recent progress in text-to-audio and text-to-video models has improved generation quality within each modality, jointly modeling them remains challenging due to incomplete and asymmetric correspondence: audio often reflects only a subset of the visual scene, and vice versa. Naively enforcing full alignment introduces semantic noise and temporal mismatches. To address this, we propose a novel framework that performs selective cross-modal alignment through a learnable masking mechanism, enabling the model to isolate and align only the shared latent components relevant to both modalities. This mechanism is integrated into an adaptation module that interfaces with pretrained encoders and decoders from latent video and audio diffusion models, preserving their generative capacity with reduced training overhead. Theoretically, we show that our masked objective provably recovers the minimal set of shared latent variables across modalities. Empirically, our method achieves state-of-the-art performance on standard T2AV benchmarks, demonstrating significant improvements in audiovisual synchronization and semantic consistency.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
Thanh V. T. Tran, N. Nguyen, Luong Tran et al.· 0 citations
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Jun Zhan, Chenchen Yang, Y. Gong et al.· 0 citations
Subject-driven and multi-element video generation are central to controllable video synthesis, but existing methods still struggle to preserve identity consistency and model complex relationships among multiple subjects. In this paper, we propose Aura, a unified framework for high-fidelity and identity-consistent video generation. To better capture scene dynamics and subject interactions, we introduce AI director-level captions that provide dense and structured descriptions of video content. We further leverage a vision-language model (VLM) with learnable queries to extract multimodal semantic features from textual and visual references, covering both global semantics and fine-grained visual cues. To bridge the representational gap between the VLM and the Diffusion Transformer (DiT), we design a two-stage alignment strategy that progressively maps VLM features into the DiT feature space. For visual conditioning, we adopt token concatenation to inject reference information directly into the generation process. To distinguish heterogeneous subject types and reduce common copy-paste artifacts, we develop a subject-aware RoPE-Shift mechanism. To further differentiate reference images of different categories, we introduce subject-aware learnable tokens. In addition, we introduce Memory Tokens to balance the training signal across examples with different numbers of reference subjects. During inference, Progressive-APG (Adaptive Prompt Guidance) further alleviates oversaturation and improves semantic alignment with user prompts. Finally, we build a high-quality video-subject image dataset through a dedicated data construction pipeline. Extensive experiments show that our method achieves state-of-the-art performance on both single-subject generation and more challenging multi-element scenarios.
Zixiang Zhou, Zhentao Yu, Yifeng Ma et al.· 0 citations
Text-only training is a popular paradigm in zero-shot video captioning, where the video distribution is not available to the model during training, leading to a cross-modal gap between the training (text-only) and the inference (video-only). Previous works attempt to bridge the gap through simple linear transformations. However, the inherent gap between text and video makes cross-modal representation space alignment insufficient, resulting in inaccurate sentences. To address this issue, we propose a novel zero-shot video captioning framework (WSV) consisting of two training stages, which first generates corresponding synthetic video latent representations via a pretrained text-to-video generation model. To strengthen the fidelity of the latent representations, we propose a polisher capable of bridging the gap between real and synthetic video distributions. Subsequently, we design a prompter that conditions GPT-2 on the polished latent representations to generate the captions in the second training stage. During inference, an input video is encoded by a pretrained 3D Causal VAE and then fed directly into the prompter, which in turn guides GPT-2 to produce the final caption. Experimental results conducted on MSVD, MSR-VTT, and VATEX datasets demonstrate that our proposed method achieves scores of 52 and 95.7 on the B@4 and CIDEr metrics, respectively.
Liangyu Fu, Junbo Wang, Yuke Li et al.· 0 citations
Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.
Yeonkyeong Lee, Hyun-Young Go, Jongmin Kim et al.· 0 citations
(English) Environmental sound conveys rich semantic and contextual information about events, objects, and spatial dynamics. However, prevailing computational approaches to environmental audio analysis, such as Acoustic Event Detection (AED), typically reduce this complexity to discrete textual labels. While effective for automated monitoring tasks, such representations oversimplify acoustic scenes and become inadequate when auditory information must be communicated across modalities. Sound-to-image (S2I) translation offers an alternative approach in which a model synthesizes images that visually depict sound-emitting sources and their surrounding environments.
This thesis introduces and advances direct sound-to-image translation, a paradigm that establishes a connection between audio and visual modalities without relying on textual mediation, class supervision, or cluster-based alignment during training. The central hypothesis is that higher-level abstractions learned by deep neural networks provide a shared semantic space in which heterogeneous modalities can connect directly, enabling the generation of images that are interpretable and semantically coherent with the source sound. Such outputs are referred to as informative, meaning that they visually communicate meaningful aspects of the acoustic event.
The first part of the thesis presents, to the best of our knowledge, the first study dedicated to direct S2I translation. A densely connected generative adversarial network (GAN), conditioned on audio embeddings, is developed to synthesize images directly from sound. Because multiple plausible images may correspond to a single acoustic event, translation quality cannot be evaluated through pixel-level reconstruction. To address this challenge, an informativity-based evaluation framework is proposed, employing classifiers to determine whether generated images are interpretable and semantically coherent with the source audio. Experiments reveal that, despite the inherent difficulty of the task, the model generalizes to unseen sounds and produces informative outputs for a meaningful portion of translations. Analysis further reveals that latent bottleneck dimensionality influences translation behavior, exposing a trade-off between pixel-space convergence and informativity.
Building on this foundation, the second part investigates whether attention mechanisms can strengthen cross-modal alignment. Self-attention and cross-attention modules are integrated into the generator and evaluated across multiple configurations. Results show that attention improves translation performance when applied at early stages of the network, increasing informativity relative to the purely convolutional baseline while preserving the fully direct audio–visual connection.
Together, these contributions formally define, validate, and extend direct S2I translation as a distinct research paradigm. Beyond architectural design and training strategies, this thesis advances methodological principles for quantitative evaluation of cross-modal translation in the absence of deterministic visual ground truth. The findings contribute to a broader understanding of multimodal representation learning and highlight the potential of direct S2I translation for applications in multimodal interaction and accessibility-oriented technologies, particularly for enhancing situational awareness in deaf and hard-of-hearing individuals.
(Català) El so ambient transmet una rica informació semàntica i contextual sobre esdeveniments, objectes i dinàmiques espacials. Tanmateix, els enfocaments computacionals predominants per a l’anàlisi de l’àudio ambiental, com la Detecció d’Esdeveniments Acústics (Acoustic Event Detection, AED), solen reduir aquesta complexitat a etiquetes textuals discretes. Tot i que aquestes representacions són eficaces per a tasques de monitoratge automatitzat, simplifiquen excessivament les escenes acústiques i resulten insuficients quan la informació auditiva s’ha de comunicar entre diferents modalitats. La traducció de so a imatge (Sound-to-Image, S2I) ofereix un enfocament alternatiu en què un model sintetitza imatges que representen visualment les fonts emissores de so i els entorns que les envolten.
Aquesta tesi introdueix i desenvolupa la traducció directa de so a imatge, un paradigma que estableix una connexió entre les modalitats auditiva i visual sense recórrer a mediació textual, supervisió per classes o alineament basat en clústers durant l’entrenament. La hipòtesi central és que les abstraccions d’alt nivell apreses per xarxes neuronals profundes proporcionen un espai semàntic compartit en què modalitats heterogènies poden connectar-se directament, permetent generar imatges que siguin interpretables i semànticament coherents amb el so d’origen. Aquestes sortides s’anomenen informatives, en el sentit que comuniquen visualment aspectes significatius de l’esdeveniment acústic.
La primera part de la tesi presenta, fins on arriba el nostre coneixement, el primer estudi dedicat a la traducció directa S2I. Es desenvolupa una xarxa generativa antagònica (Generative Adversarial Network, GAN) densament connectada i condicionada mitjançant incrustacions d’àudio, amb l’objectiu de sintetitzar imatges directament a partir del so. Atès que múltiples imatges plausibles poden correspondre a un mateix esdeveniment acústic, la qualitat de la traducció no es pot avaluar mitjançant la reconstrucció a nivell de píxel. Per abordar aquest repte, es proposa un marc d’avaluació basat en la informativitat, que utilitza classificadors dedicats per determinar si les imatges generades són interpretables i semànticament coherents amb l’àudio d’origen. Els experiments revelen que, malgrat la dificultat inherent de la tasca, el model generalitza a sons no vistos i produeix sortides informatives en una proporció significativa de les traduccions. L’anàlisi també mostra que la dimensionalitat del coll d’ampolla latent influeix en el comportament de la traducció, revelant un compromís entre la convergència en l’espai de píxels i la informativitat.
Sobre aquesta base, la segona part investiga si els mecanismes d’atenció poden reforçar l’alineament entre modalitats. Mòduls d’autoatenció i d’atenció creuada s’integren al generador i s’avaluen en múltiples configuracions. Els resultats mostren que l’atenció millora el rendiment de la traducció quan s’aplica en les primeres etapes de la xarxa, incrementant la informativitat en comparació amb la línia base purament convolucional, alhora que preserva la naturalesa plenament directa de la connexió àudio–visual.
En conjunt, aquestes contribucions defineixen, validen i amplien formalment la traducció directa S2I com un paradigma de recerca diferenciat. Més enllà del disseny arquitectònic i de les estratègies d’entrenament, aquesta tesi avança principis metodològics per a l’avaluació quantitativa de la traducció entre modalitats en absència d’una veritat visual determinista. Els resultats contribueixen a una millor comprensió de l’aprenentatge de representacions multimodals i destaquen el potencial de la traducció directa S2I per a aplicacions en la interacció multimodal i en tecnologies orientades a l’accessibilitat, especialment per millorar la consciència situacional en persones sordes o amb pèrdua auditiva.
(Español) El sonido ambiente transmite rica información semántica y contextual sobre eventos, objetos y dinámicas espaciales. Sin embargo, los enfoques computacionales predominantes para el análisis de audio ambiental, como la Detección de Eventos Acústicos (Acoustic Event Detection, AED), suelen reducir esta complejidad a etiquetas textuales discretas. Aunque estas representaciones son eficaces para tareas de monitorización automatizada, simplifican en exceso las escenas acústicas y resultan insuficientes cuando la información auditiva debe comunicarse entre diferentes modalidades. La traducción de sonido a imagen (Sound-to-Image, S2I) ofrece un enfoque alternativo en el que un modelo sintetiza imágenes que representan visualmente las fuentes emisoras de sonido y los entornos que las rodean.
Esta tesis introduce y desarrolla la traducción directa de sonido a imagen, un paradigma que establece una conexión entre las modalidades auditiva y visual sin recurrir a mediación textual, supervisión por clases o alineamiento basado en clústeres durante el entrenamiento. La hipótesis central es que las abstracciones de alto nivel aprendidas por redes neuronales profundas proporcionan un espacio semántico compartido en el que modalidades heterogéneas pueden conectarse directamente, permitiendo generar imágenes que sean interpretables y semánticamente coherentes con el sonido de origen. A tales salidas se las denomina informativas, en el sentido de que comunican visualmente aspectos significativos del evento acústico.
La primera parte de la tesis presenta, hasta donde alcanza nuestro conocimiento, el primer estudio dedicado a la traducción directa S2I. Se desarrolla una red generativa antagónica (Generative Adversarial Network, GAN) densamente conectada y condicionada mediante incrustaciones de audio, con el objetivo de sintetizar imágenes directamente a partir del sonido. Dado que múltiples imágenes plausibles pueden corresponder a un mismo evento acústico, la calidad de la traducción no puede evaluarse mediante reconstrucción a nivel de píxel. Para abordar este desafío, se propone un marco de evaluación basado en la informatividad, que emplea clasificadores dedicados para determinar si las imágenes generadas son interpretables y semánticamente coherentes con el audio de origen. Los experimentos revelan que, a pesar de la dificultad inherente de la tarea, el modelo generaliza a sonidos no vistos y produce salidas informativas en una proporción significativa de las traducciones. El análisis también muestra que la dimensionalidad del cuello de botella latente influye en el comportamiento de la traducción, revelando un compromiso entr