Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

Why it matters

OpenAI has published the second part of an engineering series explaining how its storage platform, Habitat, was scaled to serve ChatGPT users worldwide. Written by Jon Lee, Hrachya Hakobyan, and Chaomin Yu, members of technical staff, the post describes the evolution of the platform's use of Azure Cosmos DB, the database underlying its online storage layer. The authors recount a progression through distinct phases: initially relying on Azure Cosmos DB primitives, then experimenting with workload isolation and single-tenant containers, and finally settling on multi-tenancy by default with the ability to move data. They explain the tradeoffs involved, including noisy-neighbor problems, capacity planning difficulties, and partition key collisions. The post also covers performance work, including placing data near users, selective replication, integrated caching, reducing proxy overhead with Rust, and scaling change data capture. The piece is notable as a rare detailed account of infrastructure decisions behind a widely used consumer AI product, and it highlights a close partnership with the Azure team in arriving at co-managed containers as the operating model.