Case study · AI, data & growth
Building a Team-Owned Experimentation Practice
I built a team-owned experimentation practice that achieved an 11% win rate, generated €1.2M in additional revenue and uncovered a key measurement issue.
The context
Product decisions were often based on intuition or competitor benchmarks. When we ran A/B tests aimed at reducing returns, the results were hard to interpret.
The challenge
I set out to help the product teams use reliable evidence to make decisions independently, while ensuring that our experiments and quality metrics were measured correctly.
How I approached it
I ran a workshop for the product team covering why we experiment, how opinions differ from hypotheses and how evidence ranges from personal opinion to follow-up A/B tests. I introduced primary, secondary and guardrail metrics, along with common pitfalls such as peeking and sample-ratio mismatch. The teams then began writing their own hypotheses and experiment post-mortems.
When I investigated the returns experiments, I found three measurement problems: returns could arrive up to two weeks after a lead, experiment allocation happened per visitor rather than per transaction, and a small number of large partners skewed the results. I proposed attributing each return back to the date of its original lead so that outcomes aligned with the correct experiment period.
What it delivered
The software team’s experiments achieved an 11% win rate and delivered €1.2M in additional revenue. Identifying the measurement flaw also showed that returns were not a dependable quality metric, leading us to replace them with a more suitable measure.
Looking back
I learned that building an experimentation practice is as much about shared understanding as statistical technique. I valued how the teams began writing their own hypotheses and post-mortems. I would investigate the mechanics of key metrics earlier, particularly delayed outcomes, allocation levels and skew from large partners, before interpreting results.
The principle I carry forward is that evidence must be both reliable and owned by the teams using it. Clear primary, secondary and guardrail metrics create independence, but only when the underlying measurement reflects what actually happened. When a metric proves unsuitable, replacing it is more valuable than defending it.