Structured PREreview of "Training, learning and inference: unified dynamics of neural systems"

Abstract

This Zenodo record is a permanently preserved version of a Structured PREreview. You can view the complete PREreview at https://prereview.org/reviews/22899514. Does the introduction explain the objective of the research presented in the preprint? Yes The introduction clearly establishes the motivation, core research objective, and scope of the paper: Problem Context: Various computer systems mechanisms (such as database lineage, Source Maps, OpenTelemetry, W3C PROV, and PyTorch Autograd) track how results are formed. However, each field faces limitations when answering why an expected result was absent, how multi-stage transformations occurred, or how training inputs, computations, and parameter updates are causally connected. Core Research Objective: The paper aims to establish a single unified generation relation—defined as an atomic complete generation fact $f = (u, \tau, \omega, z; \rho)$—and compile these facts into a Generation-Fact Graph (GFG) to serve as an AI-native, compilable substrate for scientific provenance and fact tracking. Scope & Application: The author uses this GFG substrate and a recursive scientific process to investigate the unified dynamics across training, learning, and inference in neural systems (evaluated empirically using nanoGPT, ResNet/CIFAR, and Diffusion models). Are the methods well-suited for this research? Highly appropriate Justification: The methodology presented in the paper is exceptionally rigorous, well-structured, and uniquely suited to establishing a unified theory of training, learning, and inference in neural systems.Methodological Strengths:Formal Operational Framework: The introduction of irreducible atomic generation facts $f = (u, \tau, \omega, z; \rho)$ compiled into a Generation-Fact Graph (GFG) provides a mathematically precise, machine-verifiable substrate for tracing execution histories across multi-stage neural transformations.Causal Interventions vs. Observational Correlation: Rather than relying on simple scalar metrics (such as loss curves or accuracy summaries), the study employs matched causal forks, optimizer pauses, receiving-state exchanges, component gating (CSRG-4C), and finite-amplitude update sweeps ($\alpha \in \{0, 0.125, 0.25, 0.5, 0.75, 1\}$). This allows the author to isolate causal mechanisms from passive correlation.Prospective Out-of-Sample Validation: Theoretical findings were rigorously validated through prospective prediction on fully held-out confirmation runs, achieving 91.43% target-boundary accuracy and 91.49% macro-averaged recall without relying on post-hoc curve fitting.Cross-System Generalizability: To ensure findings were not artifacts of a single architecture, the experimental protocols were validated across structurally distinct models and optimization algorithms—specifically ResNet-18 with SGD/momentum and DDPM Diffusion U-Nets with AdamW on CIFAR datasets. Reproducibility & Open Science Best Practices: Experimental protocols, stopping conditions, and evaluation rules were frozen prior to outcome inspection. All execution histories, content hashes, Zenodo evidence archives, and independent verification scripts are fully documented and publicly available. Overall, the combination of formal state representation, prospective predictive testing, counterfactual causal interventions, and multi-architecture validation provides an exceptionally robust foundation for drawing valid scientific conclusions. Are the conclusions supported by the data? Somewhat supported Justification: The primary mechanistic conclusions drawn in the paper are well-supported by quantitative, prospective empirical data across multiple tested architectures, though validating these mechanics on multi-billion parameter frontier models remains an important direction for future work. Key Empirical Strengths: Prospective Predictive Validation: The paper's central claim—that capability transitions can be predicted via receiving states and update geometry—is backed by a second-order target-boundary predictor. On a fully held-out confirmation split, it achieved 91.43% target-boundary accuracy and 91.49% macro-averaged recall across capability state transitions (remained correct, declined, remained incorrect, recovered). Direct Causal Intervention: Component version rollbacks across 52 checkpoint pairs demonstrated in 100% of cases (52/52) that reverting a component to its pre-formation state systematically reduced performance, whereas restoring the trained component recovered exact logits. This provides direct causal proof that frozen inference depends on specific functional support built during training. Cross-Architecture Consistency: The core mechanics were validated across three structurally distinct model families—nanoGPT (transformers), ResNet-18 (convolutional networks on CIFAR-100), and DDPM Diffusion U-Nets (generative models on CIFAR-10)—confirming that the observed receiving-state dynamics are not confined to a single architecture. Constructive Bounds & Future Work: Scale Generalization: The empirical evaluation focuses on small-to-medium scale models (nanoGPT, ResNet-18, DDPM) and standard benchmarks (CIFAR-10/100). While the internal mechanics are thoroughly proven for these systems, demonstrating that these exact boundary predictors hold without distortion in multi-billion parameter frontier LLMs or multi-modal systems is a natural next step to establish universal scope. Are the data presentations, including visualizations, well-suited to represent the data? Somewhat appropriate and clear Justification: The paper's visualizations and data presentations are cleanly formatted and logically structured, though under strict peer-review evaluation, the visual presentation is overly sparse given the breadth and complexity of the experimental work. Strengths: Conceptual Process Diagram (Figure 1): The circular lifecycle diagram effectively maps the recursive scientific process, illustrating how scientific questions guide GFG analysis, intervention, replay, and recompilation across iterative cycles ($GFG_0 -> GFG_1 -> ...).Nonlinear Response Morphology Plot (Figure 2): Figure 2 clearly depicts the target response curves across normalized update amplitudes ($\alpha \in \{0, 0.125, 0.25, 0.5, 0.75, 1.0\}$), visually demonstrating the four distinct nonlinear morphologies (saturating, accelerating, turnback, and sign reversal) that falsify simple linear extrapolations. Structured Tabular Summaries (Tables 1, 2, and 3): Tables 1–3 present concise, well-labeled summaries connecting empirical tests, quantitative metrics (such as transition recall percentages and Spearman correlation coefficients), and formal theoretical implications.Weaknesses and Areas for Improvement:Absence of Visual Graph Renderings: Although the primary conceptual contribution of the paper is the Generation-Fact Graph (GFG), the manuscript contains no visual graph renderings or node-link diagrams illustrating what a materialized GFG or sub-ontology trace actually looks like in practice.Sparsity of Figures Relative to Experimental Scope: Across a comprehensive paper detailing experiments on nanoGPT, ResNet-18, DDPM Diffusion models, CSRG-4C gating, and multi-seed reinforcement learning, there are only two figures in the main text.Lack of Distribution and Trend Visualizations: Key empirical findings—such as the dose-response trade-offs in reinforcement learning, component gating distributions, and cross-system validation across ResNet and Diffusion models—are presented almost entirely through aggregated text and summary tables rather than multi-panel scatter plots, box plots, or training trajectory curves with confidence intervals. In summary, while the existing figures and tables are accurate and clear, expanding the visual representation to include actual GFG graph structures and multi-seed empirical distributions would significantly enhance readability and data comprehension. How clearly do the authors discuss, explain, and interpret their findings and potential next steps for the research? Somewhat clearly Justification: The author provides an exceptionally clear, mathematically formal, and logically deep explanation of the internal mechanics governing training, learning, and inference. However, under strict peer-review scrutiny, the paper lacks a dedicated discussion of practical limitations and explicit future research steps. Strengths in Discussion and Interpretation: Mechanistic Chain Clarity: The paper clearly articulates a step-by-step mechanistic chain connecting low-level execution actions to observable model capabilities: Training Action → Parameter–Adam Receiving State & Geometry Conditioning → Finite-Amplitude Nonlinear Response → Support Reorganization → Readout Boundary Evaluation → Observable Capability Outcome. Insightful Theoretical Connections: The author provides insightful explanations for major deep learning phenomena within the GFG framework: Why Attention Succeeds: Explains Attention as the architectural realization of query-conditioned projection and non-additive combination of learned functional support. Why Scaling Laws Hold: Connects scaling law dynamics to the expansion of projectable functional support rather than simple scalar parameter counts. Reinforcement Learning Trade-offs: Identifies a strict dose-response trade-off where concentrated positive feedback amplifies target capability support while systematically degrading unreinforced skill margins. L