Counting and Min-Cost Encoding for Tokenization in Large Language Models
Mainstream large language models rely on a tokenizer to encode text into a token sequence. Different tokenizers may yield token sequences of substantially different lengths for the same text. With a fixed model architecture, shorter token sequences correspond to lower inference time. We propose a tokenizer training app...
Shu-Ming Shi, Xiang Zhang, Hao-Yi Yu et al.
· 0 citations