JTCSE: Joint tensor-modulus constraints and cross-attention for unsupervised contrastive learning of sentence embeddings.

Abstract

Unsupervised contrastive learning has attracted significant attention in natural language processing for learning sentence embeddings. Existing methods primarily constrain the orientation distribution of representations in high-dimensional semantic space via cosine-based objectives, yet the embedding vector also possesses a magnitude (norm) property. We observe that ignoring magnitude consistency between positive-pair representations limits the effectiveness of contrastive learning. To address this, we propose a magnitude-alignment training objective that explicitly constrains the norm discrepancy between positive-pair embeddings, strengthening contrastive alignment from both directional and magnitude perspectives. Furthermore, BERT-like encoders suffer from the attention sink phenomenon, whereby disproportionate attention mass is allocated to non-informative tokens at the expense of the [CLS] token that aggregates sequence-level semantics. To mitigate this, we introduce a cross-attention mechanism within a dual-encoder architecture that enhances the model's focus on [CLS] representations and improves the quality of CLS-based pooling. Building upon these two complementary ideas, we present JTCSE (Joint Tensor representation modulus constraint and Cross-attention unsupervised contrastive learning Sentence Embedding framework). We evaluate JTCSE on seven semantic textual similarity benchmarks with controlled multi-seed experiments and statistical significance tests; both the dual-encoder ensemble model and its single-encoder distilled variant achieve competitive or state-of-the-art performance. Extensive zero-shot evaluation across more than 140 downstream tasks further demonstrates that JTCSE consistently outperforms existing baselines. Source code is available at https://github.com/tianyuzong/JTCSE, saved checkpoints are available at https://huggingface.co/UCASzty.