Skip to content
Preprint

Integer-Exact Microscaling: Running MXFP4 Language Models on INT8 Matrix Units

Oct 2026 · 0 citations · 45 references
Computer Science

TL;DR

This work shows when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output.

Abstract

On hardware whose fastest matrix datapath is INT8$\times$INT8, such as the Neural Accelerators of Apple's M5 GPUs, 4-bit weights must become INT8 operands. Existing MXFP4 kernels already map FP4 (E2M1) values to integers by doubling them, but apply every 32-element block scale inside the reduction. We show when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output. Enforcing the condition adds at most 1/16 of a row's largest rounding error to the blocks it changes, and real weights already satisfy it: 97.9-99.97% of rows across seven models, including 99.93% of the 95M rows of a 125B-parameter mixture-of-experts (MoE) model, Qwen3.8-Flash-Next. The resulting spread-constrained MXFP4 files stay standard MXFP4. With integer instead of power-of-two block multipliers, an integer-exact format (16-wide blocks, multipliers 1..10, 4.26 bits) runs as one exact INT8 GEMM and is more accurate than NVFP4 (4.5 bits) with its standard scale rule on five small models, within 0.06 perplexity of NVFP4 with scale search. On Qwen3.8-Flash-Next's routed experts with INT8 activations it is as close to the FP8 model as NVFP4 with scale search (KL divergence 0.0557 vs. 0.0553) and closer than MLX's MXFP4 and affine INT4; with every matrix quantized it solves 94.8% of GSM8K vs. 91.0% for MLX's MXFP4. Quantizing only the 74k routed expert matrices (96% of the parameters) to either of our formats with INT8 activations costs at most 0.2% perplexity. A complete W4A8 system on the M5 produces outputs bit-identical to the quantized reference; prefill is 1.34-1.74x faster than MLX on a dense model, 1.10-1.57x on two MoE models up to 35B parameters and 1.22-1.29x on Qwen3.8-Flash-Next, where shared non-GEMM work bounds the gain.

View source

Similar papers

Preprint Oct 2026

Investigating ET-SoC-1 Vector and Parallel Performance with sGEMM

The Ainekko (formerly Esperanto) ET-SoC-1 is a massively parallel RISC-V accelerator. Its compute array comprises 1024 dual-threaded in-order ET-Minion cores (2048 harts) operating at 600 MHz, each core providing an eight-lane 256-bit packed-single (PS) SIMD datapath shared by its two harts, with access to LPDDR4X memo...

Richard Neale, Nick Brown · 0 citations
Preprint Oct 2026

Deciding Scale Placement in MXFP4 Dot Products with Integer Arithmetic

The Open Compute Project's microscaling (MX) family includes MXFP4, which stores 32 four-bit elements with a shared power-of-two scale. When lowering a dot-product reduction, a compiler can apply these scales before the reduction, or defer scaling until afterwards. Although equivalent over real numbers, these formulati...

Shreyas Vinayakumar · 0 citations
#machine learning Preprint Nov 2026

ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4

ZYFA, prefill kernels for the MI355X built on three techniques: computing each FP8 probability directly from its score with a multiply-add and an integer conversion instead of an exponential; a ping-pong schedule in which one group of threads computes its softmax while another executes its matrix multiplications; and s...

Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli et al. · 0 citations
Preprint Nov 2026

SFU-Sim: Bit-Exact Reverse Engineering of NVIDIA MUFU and Validation of PTX Paths

Reproducing the numerical behavior of a GPU requires predicting the exact output of special function instructions. Vendor documents give only the function semantics and error bounds. These do not fix the output bit pattern. We present SFU-Sim, which builds bit-exact models of NVIDIA multi-function unit (MUFU) instructi...

Hao-Zhi Xie, Tao Su · 0 citations
Preprint Oct 2026

metalSPARSE: A 2:4 Semi-Structured Sparse GEMV Kernel for Apple Silicon GPUs

Autoregressive Large Language Models (LLMs) generate text one token at a time, and each token reads every weight from memory, so on edge devices, decoding is limited by memory bandwidth. 2:4 semi-structured sparsity keeps two of every four weights, but Apple-silicon GPUs have no sparse matrix hardware, so pruning helps...

Tek Raj Awasthi, Babita Awasthi · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.