The Discrete-Log Clock: How a Transformer Learns Modular Multiplication
Huu Danh Nguyen
Aug 2026
Artificial IntelligenceMachine Learning
Abstract
When small transformers grok modular multiplication, prior work reports that the learned embedding has a "dense" Fourier spectrum requiring all frequencies. This contrasts with modular addition, where only a sparse set of key frequencies suffices. We show this density is an artifact of analyzing in the wrong basis. The natural Fourier transform for multiplication is not the standard additive DFT but the multiplicative character transform, which decomposes functions on the multiplicative group $(\mathbb{Z}/p\mathbb{Z})^*$ into its irreducible representations. Applying this transform to a grokked transformer trained on $a \cdot b \bmod 113$, we find the embedding spectrum becomes highly sparse (Gini coefficient 0.58 vs. 0.07 in the additive basis) with only 4 key frequencies carrying significant energy. Furthermore, 96.9% of MLP neurons are cleanly tuned to a single multiplicative frequency, and neuron activation heatmaps reveal 2D-periodic structure when reordered by the discrete logarithm. These results demonstrate the transformer reduces multiplication to addition in discrete-log space, implementing a "Discrete-Log Clock" algorithm analogous to Nanda et al.'s Clock algorithm for addition. The methodology generalizes: matching the analysis basis to the algebraic structure of the task reveals interpretable structure where standard tools see noise.
This article investigates several physics-informed and hybrid machine learning strategies that incorporate physics knowledge in experimental data-driven deep-learning models for predicting the bond quality and porosity of fused filament fabrication (FFF) parts. Three types of strategies are explored to incorporate physics constraints and multi-physics FFF simulation results into a deep neural network (DNN), thus ensuring consistency with physical laws: (1) incorporate physics constraints within the loss function of the DNN, (2) use physics model outputs as additional inputs to the DNN model, and (3) pre-train a DNN model with physics model input-output and then update it with experimental data. These strategies help to enforce a physically consistent relationship between bond quality and tensile strength, thus making porosity predictions physically meaningful. Eight different combinations of the above strategies are investigated. The results show how the combination of multiple strategies produces accurate machine learning models even with limited experimental data.
B. Kapusuzoglu, S. Mahadevan· JOM· 79 citations· ⚡2
This work introduces a pioneering exploration of Self-Supervised Learning (SSL) within the SNN, and proposes a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer) that achieves 80+% accuracy on ImageNet.
Zhaokun Zhou, Kaiwei Che, Wei Fang et al.· arXiv.org· 69 citations· ⚡10
This paper considers global sensitivity analysis (GSA) for situations where both a physics-based model and experimental observations are available, and investigates physics-informed machine learning strategies to effectively combine the two sources of information in order to maximize the accuracy of the sensitivity estimate.
B. Kapusuzoglu, S. Mahadevan· Reliability Engineering & Sy...· 45 citations
EquiPocket is proposed, an E(3)-equivariant Graph Neural Network for binding site prediction, which comprises three modules: the first one to extract local geometric information for each surface atom, the second one to model both the chemical and spatial structure of protein and the last one to capture the geometry of the surface via equivariant message passing over the surface atoms.
Yang Zhang, Wenbing Huang, Zhewei Wei et al.· International Conference on...· 43 citations· ⚡4
Investigating how experienced developers use agents in building software, including their motivations, strategies, task suitability, and sentiments finds that while experienced developers value agents as a productivity boost, they retain their agency in software design and implementation out of insistence on fundamental software quality attributes.
An adaptive surrogate modeling method for problems with very high-dimensional spatio-temporal outputs is developed that combines exploration and exploitation to improve the surrogate model accuracy with the fewest possible runs of the expensive physics-based model.
B. Kapusuzoglu, S. Mahadevan, Shunsaku Matsumoto et al.· Structural And Multidiscipli...· 17 citations
