Khubaib Ahmed

Shift-Accumulate Attention: Multiplier-Free Query--Key Products for Transformer Decoding

Power-of-two (PoT) quantisation turns a multiplication into a bit shift, so far only for the post-softmax attention--value product. The earlier and larger product, S=QK^T, has not been reformulated the same way. We quantise the key cache to a signed power-of-two fixed-point code, so that every scalar multiplication in...