Skip to content
Preprint

Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production

Aug 2026 · 0 citations · 73 references
Computer Science

TL;DR

Uni-SLTP is proposed, a unified framework for SLT and SLP with two key components: a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information.

Abstract

Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.

View source

Similar papers

#computer vision Open access Sep 2026

SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale

To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data, which shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to si...

Zhaoyi An, Si-Han Tan, Youngbae Hwang et al. · 0 citations
#large language models Review Oct 2026

Machine Translation for Sign Languages

Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation systems. Advances in pose estimation, transformer architectures, and large-scale dataset collection have driven progress, yet challenges remain. Datasets are limited com...

Ozge Mercanoglu Sincan, A. Pelykh, Edward Fish et al. · 0 citations
Sep 2026

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

SignGPT is presented, a unified, pose-based framework for gloss-free SLT and SLG that integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling.

Rong-Hui Li, Jun Dong, Zhong-Yuan Hu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks

This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation that aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4.

Oline Ranum, Edward Fish, Simon Hadfield et al. · 0 citations
Open access 2026

Bridging Text-to-Sign Translation via Codebook-Oriented Pretraining

This work proposes a novel text-to-sign translation based on model pretraining, which enhances semantic alignment by inheriting codebook-oriented prior knowledge from masked self-supervised models.

Ninlawat Phuangchoke, C. Polprasert · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.