This talk covers how to design and run holdout experiments based on practical experience at Deliveroo. Standard A/B tests are optimised for speed — but they systematically miss effects that only emerge over time or through the interaction of multiple features. Holdouts are a tool for closing that gap. The talk covers patterns used at Deliveroo, ranging from single-feature holdouts that extend measurement beyond the experiment window, to multi-feature holdouts that capture cumulative effects across a related set of changes, to multi-team holdouts that track progress against a shared business goal. It also covers the practical challenges that arise when running holdouts in a complex, experiment-heavy environment — from design decisions made before launch to the difficulties of adapting when reality does not go to plan.
Technical Level of Session: Technical practitioner
Supported by
