Impact Analyzer
Did your new feature improve conversion? Compare a Control group with a Treatment group, quantify the effect, and learn how to interpret statistical significance. All calculations run locally in your browser.
1. What are you comparing?
Control represents the original experience (for example, the existing app version). Treatment represents the changed experience (for example, a new onboarding flow). A conversion is the action you want users to complete.
2. Enter your experiment data
Control · Original version
How many unique users experienced the original version?
How many of those users completed the goal?
Treatment · New version
How many unique users experienced the new version?
How many of those users completed the goal?
Ideally choose this before looking at results.
3. Experiment results
4. Learn how to interpret the metrics
Conversion rate (CVR)
CVR = converted users ÷ exposed users. It measures the proportion of exposed users who completed the target action.
Absolute difference vs. relative uplift
Absolute difference = Treatment CVR − Control CVR, expressed in percentage points. Relative uplift = (Treatment CVR − Control CVR) ÷ Control CVR. A move from 10% to 12% means +2 percentage points and +20% relative uplift.
What does the p-value mean?
The two-sided p-value describes how unusual a difference at least this extreme would be if the two underlying conversion probabilities were equal, under the test's assumptions. It is not the probability that the null hypothesis is true.
What is statistical significance?
When p < α, the observed difference is called statistically significant. This does not guarantee that the effect is important in practice or caused by the feature. Random assignment, data quality, appropriate stopping rules, and measurement consistency matter.
What does a confidence interval tell us?
A confidence interval provides a range of effect sizes compatible with the data under a statistical procedure. This demo uses a simple Wald interval, which can perform poorly for small samples or rare conversions. Do not use it as a substitute for a robust production analysis.
When should I avoid the Z-test?
The normal approximation can be unreliable when expected successes or failures are small. This demo flags groups with fewer than five successes or failures. For sparse data, consider Fisher's exact test or other appropriate methods.