Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Tax
A token-cost ledger is assembled that splits each language's cost, at fixed parallel content, into a removable coding redundancy, a residual coding slack, an intrinsic-content term, and an orthogonal, irreducible grapheme-to-phoneme term that governs the multimodal rather than the text cost.
Madhulatha Mandarapu, Sandeep Kunkunuru
· 0 citations