If you remember Chapter 4 of Trustworthy Online Controlled Experiments—the one with the blue HIPPO on the cover and the crawl–walk–run–fly maturity model—this session brings that chapter to life. It tells the story of how we intentionally progressed our experimentation platform from “walk” to “run,” navigating build-vs-buy trade-offs, architectural constraints, and organizational realities along the way. Rather than focusing on building custom tooling, we made deliberate decisions to adopt a modern, market-leading experimentation platform while investing internally in the foundations that truly enable scale: governance, standards, and culture.
The session starts by revisiting the core principles of trustworthy experimentation—drawing from established literature and industry practices—to ground the discussion in what actually matters when running experiments at scale. From there, it reflects on the capabilities of a long-standing in-house platform, the value it unlocked over time, and the natural limits that eventually made it a bottleneck for further maturity. Attendees will then learn how we evaluated, selected, and integrated a third-party experimentation platform, and how that decision unlocked a virtuous cycle of progress across technology, data practices, governance, and experimentation culture. The talk concludes with early signals and lessons from running production-grade experiments on the new platform, offering a realistic and actionable view of what “moving from walk to run” looks like in practice.
Technical Level of Session: High Level/overview
Supported by
