Aug 2026· International Conference on Foundations of Digital Games· pp. 1-9· 0 citations· 6 references
Computer Science
TL;DR
A group-supervised contrastive learning framework that addresses semantic ambiguity through two complementary mechanisms that expands parameter-based templates into diverse natural language descriptions using Large Language Models, and a group-supervised multi-positive contrastive objective that aligns text embeddings with sets of maps generated under shared control parameters is introduced.
Abstract
Learning general-purpose representations for symbolic procedural content generation (PCG) remains challenging due to the domain gap between discrete tile maps and natural images, and the lack of reusable semantic backbones for analyzing generated content. A key challenge in PCG is that diverse maps may share identical design intent when generated from the same control parameters, creating many-to-one semantic mappings. Standard contrastive learning approaches struggle in this setting, often treating semantically equivalent maps as negatives, leading to fragmented embedding spaces. In this paper, we present a controlled study on learning semantic representations for symbolic maps aligned with natural language. We introduce a group-supervised contrastive learning framework that addresses semantic ambiguity through two complementary mechanisms: (1) a data construction pipeline that expands parameter-based templates into diverse natural language descriptions using Large Language Models (LLMs), and (2) a group-supervised multi-positive contrastive objective that aligns text embeddings with sets of maps generated under shared control parameters. We evaluate our approach in a Zelda-based grid environment against visual (CLIP, ResNet) and statistical (Tile-MLP) baselines. Results show that explicitly modeling semantic groups leads to improved retrieval accuracy (60.8% R@5 versus 25.7% for statistical baselines), with larger gains under out-of-distribution settings. Moreover, the learned representations support zero-shot inference of several abstract gameplay attributes (e.g., solvability and threat level) within tile-aligned domains, without task-specific supervision. While cross-game transfer remains limited and sensitive to tile mapping quality and domain shift, these results suggest that parameter-guided semantic alignment is a practical step toward reusable representation backbones for symbolic PCG analysis within related domains.
Understanding a painting is never a single act. Art historians may analyze the same work through concepts of style, iconography, or historical context, dimensions that are not interchangeable, and each carries distinct semantic relationships between the visual and the textual. Vision-Language Models (VLMs) like CLIP, which learn a single shared embedding space, collapse this richness into a single homogeneous alignment, thereby losing the multi-relational structure that defines art-historical reasoning. We introduce CANVAS (Contrastive Art-aware Network for Vision-Language Alignment with Sheaves), a framework for learning relation-aware multimodal representations inspired by sheaf theory. Each artwork is projected into multiple embeddings conditioned on the type of relation (i.e., the context), and a novel contrastive loss encodes contextual information during training, with no dependency on external data at inference. We evaluate on three newly introduced benchmarks of artworks for multi-relational art understanding: WikiArt+, derived from WikiArt and Wikipedia, HertzianaDP, from the Bibliotheca Hertziana collection, and SemArt+, refined from the SemArt dataset. In multimodal retrieval and art understanding, CANVAS outperforms the baselines, supporting the view that multi-relational alignment is not just theoretically motivated but also practically essential.
Ludovica Schaerf, Antonio Purificato, Piera Riccio et al.· 1 citation
Vision-Language Models (VLMs) have achieved remarkable success across various downstream tasks. However, their compositional understanding of complex attributes and relations remains a significant challenge. While existing methods leverage compositional datasets and contrastive learning, they are often compromised by various interferences such as textual ambiguity and synthetic artifacts. To address these issues, this paper proposes DC-CLIP, a framework designed to extract and leverage fine-grained semantic primitives from image-text pairs. Specifically, a Semantic Decoupling Module (SDM) is introduced to decompose global inputs into fine-grained entities and their corresponding local regions. Then, these primitives are fed into the encoders alongside the original data, with feature integration facilitated by a Feature Coupling Module (FCM). Furthermore, a Decoupling Contrastive Loss (DCL) is proposed to strengthen the representation learning of critical semantic features. Extensive experiments on ARO and SugarCrepe demonstrate that DC-CLIP significantly outperforms state-of-the-art methods in compositional understanding tasks.
Wenxiang Chen, Housheng Su· IEEE Signal Processing Lette...· 0 citations
Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.
David Ming Segura, Jeremy Goumaz, Joshua W. Sin et al.· 0 citations
Modern language models generally represent text as sequences of discrete token embeddings, an assumption deeply rooted in current practice but rarely questioned. We challenge this representation, especially for Chinese, by replacing index-based token embeddings entirely with a single rasterized image of the character sequence, processed by a vision encoder composed of a shared ResNet and a shallow Vision Transformer. To isolate the role of input representation, we construct a dual-branch controlled framework in which both a Vision-based model and an index-based baseline share an identical decoder backbone, training objective, optimizer, and data curriculum. Any performance difference is therefore attributable to the input modality only. Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline, reaching a peak accuracy of 0.429 versus 0.355 for the index-based baseline,that is, a 21% relative improvement, while converging in about half the number of training epochs. The advantage emerges especially within the first five epochs (under 21% of total data) and persists under moderate character corruption: the corrupted Vision model matches the clean index-based baseline. Ablation studies reveal that the advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings. A cross-script comparison on English shows the advantage does not transfer directly to alphabetic writing systems, suggesting that the uniform visual density and radical structure of Chinese characters are enabling conditions. These findings suggest that transformers are more modality-agnostic than commonly assumed, and that discrete tokenization is not a fundamental requirement for Chinese language modeling.
Vision-language models pretrained on large-scale datasets have become the foundation of cross-modal understanding. However, they often overemphasize salient objects and underperform in compositional reasoning that additionally requires fine-grained perception of attributes and inter-object relations. Although compositional data-driven approaches have shown promise in addressing this challenge, low-quality captions undermine robust alignment and consequently hinder fine-grained perception. Specifically, two fundamental issues arise: (1) low-density positive captions restrict the global representation capability of image encoders, causing local alignment. (2) false-negative captions introduce noise into training, resulting in fragile alignment. In response, we propose the Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP), enhancing both the robustness and granularity of vision-language alignment. To address the first challenge, RoFLIP employs the Densified Ranking Loss (DRL), which promotes global visual representations by contrasting dense and sparse captions. For the second challenge, RoFLIP incorporates Denoised Cross-modal and Text-modal Contrastive Losses (DCCL &DTCL), reducing the impact of noise while effectively leveraging negative captions. In addition, we introduce RoFLIP’s Auxiliary dataset (RoFA), comprising 225K high-quality captions that are fluent and image-related to support training. Quantitative evaluations demonstrate that RoFLIP-CLIP-B/32 achieves an average gain of 7.41%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.41\%$$\end{document} across 5 compositional reasoning benchmarks while maintaining considerable performance on 23 downstream benchmarks. Furthermore, replacing the corresponding modules in LLaVA-v1.5 and Stable Diffusion v1.5 with RoFLIP-CLIP-L/14 also brings performance gains on VQA and image generation tasks, respectively. These results underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuanbin Liu, Shancheng Fang et al.· International Journal of Com...· 0 citations
Generating SVG-based fonts requires Multi-modal Large Language Models (MLLMs) to translate high-level linguistic intent into low-level, topologically constrained symbolic sequences. However, current approaches struggle with two fundamental misalignments: the semantic ambiguity of unstructured natural language for precise geometric control, and the inefficiency of generic text tokenizers , which fragment coordinate-dense SVG XML into excessively long sequences with low information density. In this work, we propose Vector Cal-ligrapher , a system that treats SVG generation as a conditional language modeling task optimized for both semantic grounding and representational efficiency. To bridge the semantic gap, we introduce a structured linguistic supervision Font Description Framework that decomposes typo-graphic style into interpretable linguistic dimensions (e.g., historical lineage, affective metaphors), providing structured supervision aligned with the compositional syntax of SVG. To address the tokenization bottleneck, we de-sign a scalable separated-coordinate strategy that bypasses the vocabulary explosion of flattened tokens while significantly compressing sequence length. Supported by Vec-torFont , a dataset of over 10 million hierarchically annotated glyphs, our approach improves CLIP score by +23%, reduces geometric error by ≈48%, and boosts generation efficiency by achieving an 18% Commands-per-Token (C/T) ratio—a 6 × increase in information density over standard baselines. These results demonstrate that combining structured linguistic supervision with efficient symbolic tokenization is essential for reliable, controllable vector graphics synthesis. VectorFont dataset, Code and model weights will be publicly released.
Bo Zhou, Xikang Chen, Yan Gong et al.· Annual Meeting of the Associ...· 0 citations