Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings, and proposes SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens...