Skip to content

OPEN-1B: A Fully Auditable Training Run

Sep 2026 · 0 citations · 40 references
Computer Science

TL;DR

Open-1B, a model trained under this regime, is released, together with its full pretraining dataset, every intermediate checkpoint, the training codebase, and the audit harness needed to reproduce and verify any step of its training.

Abstract

Open-source language models have a reproducibility problem. Despite releasing weights, training data, and recipes, none of them are provably reproducible due to the non-associativity of floating-point arithmetic. Deep learning frameworks often offer a deterministic execution mode, allowing reproducible operations on the same machines. Unfortunately, this determinism does not carry across hardware such that a user can verify that a released checkpoint was actually produced using the declared training recipe. This leaves room for undisclosed data, injected biases, or backdoors that existing techniques such as proof-of-learning or proof-of-training-data cannot rule out. We introduce a new tier of model transparency, fully auditable, in which every operation on every data sample during training is independently reproducible on heterogeneous commodity hardware with bitwise certainty. By imposing a definite order on the sources of training nondeterminism, GPU kernel reductions, data batch ordering across a data-parallel cluster, and inter/intra-node collective communication, we make it possible to replay any individual step of a large, distributed training run on a single piece of commodity hardware and check it against the published trajectory. Because replaying an entire run on one machine is infeasible, we support this with a collective verification scheme in which many independent auditors each certify individual steps, together covering the whole run. We release Open-1B, a model trained under this regime, together with its full pretraining dataset, every intermediate checkpoint, the training codebase, and the audit harness needed to reproduce and verify any step of its training.

View source

Similar papers

Preprint Sep 2026

Towards TEE-Certified DP: Verifiable Differentially Private Training on Legacy GPUs

Wide adoption of machine learning has created growing policy and regulatory demand for protecting sensitive training data, with differential privacy (DP) emerging as a key mechanism. Yet a less-studied problem is how to certify the faithful execution of DP during training: an external verifier should be able to check t...

Li Ge, Wenjie Qu, Weitao Feng et al. · 0 citations
#machine learning Preprint Sep 2026

Numbat: Building and Verifying a Self-Contained Machine-Learning Stack

This work reports on the construction and verification of numbat, a machine-learning stack written in one general-purpose language (Zig) with no third-party runtime dependencies, and treats a widely used reference implementation as an executable specification and verify against it at five levels.

Thang Tran, Lan Dang · 2 citations
#small language model Preprint Aug 2026

A Reproducible, License-Aware Distillation Recipe for CPUDeployable Safety Classification

A reproducible, license-aware knowledge-distillation recipe addressing the constraint of deploying a safety layer for large language models on commodity hardware by partitioning the corpus into seven safety categories aligned to a public hazard taxonomy.

Edson Rodrigues da Cruz, Paulo Ricardo Ferreira Neves, P. H. Falsetti et al. · 0 citations

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

This work presents a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its...

Liam Cooper, Shinnung Jeong, Hyeran Jeon et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.