Experiment Backlogs, Done Properly
A prioritized list of experiments waiting to run, with hypotheses and expected impact.
Samir Haddad
The key figures
- Hypotheses
- each experiment needs a stated expectation
- Prioritization
- impact, confidence and effort are common criteria
- Traffic
- test locations need enough volume
- Learning
- losing tests still produce knowledge
Why this is worth getting right
Testing programs stall without a backlog, and unprioritized backlogs test trivial changes on low-traffic pages.
Do this, not that
Do
- Write hypotheses before prioritizing
- Score by expected impact and effort
- Test on pages with enough traffic
- Record results including losses
- Review the backlog regularly
Don’t
- Testing button colors on low-traffic pages
- Backlogs with no scoring
- Results recorded only when positive
- Ideas tested without hypotheses
When to bring in help
Our advice Bring in help when testing programs lose momentum.
Where this comes from
- Nielsen Norman Group — A/B testing
- National Institute of Standards and Technology — Engineering Statistics Handbook
The figures and practices above come from the sources listed.
Working on something like this?
We take on Digital Marketing & CRO work for teams who want it done once, properly. Tell us what you are building and we will tell you honestly whether we are the right studio for it. Start a project.
Where to go next
Spotted something wrong? Report an error on this page. We correct on the page and say what changed.
