Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models
In cost-matched pairs, extra orders beat extra bits only when the weights have 6 or more bits, and the decision table reduces to: use 6-bit weights and spend the rest of the budget on up to four orders, one pass each.
Mohammed Nayeem Khan, L. Logesh, Aliyas Ahmad Misri
· 0 citations