How AI products are actually built at US Big Tech companies: approaches, mistakes, and the new role of the PM

Abstract

AI products break the assumptions most product management practice is built on: deterministic outputs, stable specs, and a clean line between engineering and design. Drawing on direct experience building AI features inside US Big Tech companies, this talk examines how product development actually happens once a model enters the stack, and where teams go wrong. We look at recurring mistakes: shipping on demo quality instead of eval quality, treating prompts as throwaway code, and letting model releases dictate the roadmap instead of user needs. We also trace how the PM role itself is shifting: from writing specs to defining evaluation criteria, from sole roadmap owner to translator between research, safety, and engineering, and from planning in quarters to iterating in days. The talk closes with a practical framework, three habits any PM can adopt to work effectively with AI, regardless of company size. Aimed at PMs, engineers, and anyone shipping AI features, it trades slogans for the operational detail of how this work actually gets done.