Almost Free State Prediction Separation
This paper takes state--prediction separation to its limit with a free pause token: a prediction stream that writes no keys or values at all and so rides the sequence's existing positions, and reduces the raw flops required at inference time.
John Langford, Nathan Godey, Giovanni Monea et al.
· 1 citation