Diagnosing and Correcting Geometric Bias in Diversity-Driven Code Instruction Data Selection

Abstract

Diversity-driven data selection offers a scalable way to reduce code instruction-tuning costs, yet it relies on a fragile assumption: geometric spread in representation space should correspond to training-useful semantic coverage. In code data selection, our diagnosis reveals a geometric bias behind this assumption: dense embedding geometry can be strongly influenced by rare surface forms, verbose constraints, and implementation artifacts, causing diversity objectives to over-explore locally sparse or globally peripheral samples whose training utility is not guaranteed. To correct this bias, we propose Semantically Anchored Geometric Coverage (SAGC), a framework for reshaping representation geometry using LLM-generated macro-intent labels as semantic anchors for dense micro-context representations. Rather than replacing standard diversity objectives, SAGC reshapes the geometry exposed to them, using macro-intent anchors to constrain dense-space exploration toward high-level algorithmic structure without requiring downstream validation signals, gradient feedback, or evaluation results. Using only 20k samples selected from a 446k pool, SAGC reaches 0.4006 on LiveCodeBench, approaching full-data training at 0.4027. Further diagnostics and out-of-benchmark evaluations show that semantic anchoring provides a structured correction to this geometric bias.