BigCodeBench: The Next Generation of HumanEval
More from the blog
We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
Our newest music generation model, Lyria 3.5, delivers significant advancements across musicality, lyrics, and vocal quality, empowering you to craft richer tracks. We’r…
Empowering India’s next generation of innovators with ATL Saathi
Google and AIM launched ATL Saathi, a Gemini-powered AI tool empowering Indian educators in robotics labs.
MIT in the media: Innovating and educating for the next 250 years of America
During a "Washington Post Live" panel discussion with ASU President Michael Crow, President Sally Kornbluth explored how universities are preparing the next generation of scientists to lead in America’s rapidly changing technological landscape.
Related papers
Constitutional Midtraining: Content Presence Drives Alignment Gains
Post-training alignment is often shallow, eroding under fine-tuning. It remains untested as to whether constitutional midtraining interventions can produce durable alignment when cleanly isolated from post-training. We build a 394M-token constitutional corpus from Anthropic's Constitution and apply constitutional midtraining at 120B scale, where principled, values-based content is inserted into midtraining. A 2x2 design (curriculum ordering x deliberative reasoning) was used to produce four constitutionally midtrained conditions, plus a control, which were evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment. All models were evaluated across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperformed the control on alignment generalization and durability, notably on blackmail: SFT instilled a blackmail propensity in all models, but constitutional midtraining blunted it, with the advantage surviving benign fine-tuning (-17.5pp). This durability did not extend to settings that required active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also mattered more than its structure, and constitutional midtraining incurred no capability cost, on average, at any stage (MMLU, ARC-Easy, piqa, GSM8K). A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available.
Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025
Investigating how experienced developers use agents in building software, including their motivations, strategies, task suitability, and sentiments finds that while experienced developers value agents as a productivity boost, they retain their agency in software design and implementation out of insistence on fundamental software quality attributes.
Unlocking non-model organisms with CRISPR-Cas: A roadmap for sustainable biotechnology.
It is concluded that bridging the gap between foundational CRISPR research and its real-world applications is imperative and future efforts should focus on democratizing tools via open-source platforms, advancing delivery systems, and fostering sustainable innovation through synthetic biology integration to fully realize the transformative potential of genome editing in organisms beyond model organisms.
Mutation-induced phenotypic plasticity drives formation of a pro-metastatic tumour-intrinsic niche
It is argued that formation of a tumour-intrinsic niche is a prerequisite for BRAF-mutant CRC seeding to distant organs and that interference with niche formation may help avoid metastatic relapse.