For more background on the *Hypothesis Kit and Power Analysis* read my Medium Post.

Based on [quantitative/qualitative insight].

We predict that [product change] will cause [impact].

We will test this by assuming the change has no effect (the null hypothesis) and running an experiment for [X week(s)].

If we can measure a Y% statistically significant change in [metric] then we reject the null hypothesis and conclude there is an effect.

The number of people visiting your site that *could* see the change on a given day.

This is the percentage of your eligible visitors that currently complete your goal (e.g. purchasing or signing up).

You need to run your experiment for at least 1 full business period.

This is the percent by which you predict you will change your base conversion rate. Use the minimum that would be worth the effort, or play with different values.

If you are 95% confident in a result, this implies that with 5% probability the observed difference is in fact due to chance.

The Hypothesis Kit was developed by Rik Higham and Colin McFarland, with contributions from David Pier, Lukas Vermeer, Ya Xu and Ronny Kohavi. “Design like you’re right, test like you’re wrong” props to Jane Murison and Karl Weick. Original Hypothesis Kit from Craig Sullivan.

Power analysis calculation based on Experiment Calculator by Dan McKinley, adapted by Rik Higham.

Copyright © Rik Higham 2016 - 2017