Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalization
It is argued this regime is a scientific instrument with four capabilities that approximate settings cannot offer: exact, falsifiable generalization ceilings; task surgery that manipulates one structural variable while provably fixing all others; direct observation of every weight; and survival-time statistics over man...