DeMER: Decoupling Character's Prototype and Style for Handwritten Mathematical Expression Recognition
Abstract
Handwritten mathematical expression recognition (HMER) focuses on automatically generating LaTeX sequences from provided images. Currently, attention-based encoder-decoder models are extensively adopted for this task. However, due to the ever-changing nature of handwriting styles, existing decoding methods, which merely learn character features from formulaic images, struggle to accurately classify similar symbols, even misclassifying the same symbol into different categories. Attention drift during character recognition further exacerbates visual confusion between characters. To tackle these problems, we propose DeMER, a unified framework integrating feature denoising and attention calibration for effective mathematical symbol representation learning. In particular, we first design an Implicit Variational Prototype Module (IVPM), which effectively decouples of prototype and style by modeling character features as Gaussian probability distributions via variational inference. Variational sampling and prototype alignment mechanisms force noisy features to map onto a standardized prototype space. Second, we introduce the Soft Redundancy Aware Coverage (Soft-RAC) attention refinement module for more precise attention localization. In addition, IVPM can be seamlessly integrated into existing RNN-based models for HMER and consistently deliver performance improvements. Experimental results on widely-used benchmarks demonstrate that our method significantly outperforms existing state-of-the-art approaches in both recognition accuracy and structural robustness.