Explanation of the Statistical Significance Calculator
The statistical significance calculator helps determine if the difference in conversion rates between two groups (control and variant) is statistically significant. This means it checks if the observed difference is likely due to a real effect rather than random chance. We also have a Bayesian Statistical Significance Calculator that calculates probability.
Here’s how it works:
- Input Data: You provide the number of sessions and conversions for both the control group and the variant group.
- Control Group: The original or default group.
- Variant Group: The group with the new changes you want to test.
- Calculate Conversion Rates:
- Conversion Rate: The proportion of sessions that result in conversions.
- For each group, the conversion rate is calculated as:
Conversion Rate = Conversions / Sessions
- Compute Standard Errors:
- The standard error measures the accuracy of the conversion rate. It depends on the conversion rate and the number of sessions.
- The formula for the standard error is:
Standard Error = sqrt((Conversion Rate * (1 - Conversion Rate)) / Sessions)
- Calculate the Z-Score:
- The z-score measures the difference between the two conversion rates relative to the combined standard error.
- The formula for the z-score is:
z = (Variant Conversion Rate - Control Conversion Rate) / sqrt(Standard Errorcontrol2 + Standard Errorvariant2)
- Determine the P-Value:
- The p-value indicates the probability that the observed difference is due to random chance.
- A low p-value (typically < 0.05) suggests that the difference is statistically significant.
- The p-value is calculated from the z-score using the cumulative distribution function of the normal distribution.
- Output the Result:
- The calculator displays the conversion rates for both groups, the z-score, the p-value, and whether the difference is statistically significant.
Summary
By using this calculator, you can confidently determine if the changes you’ve made in the variant group have led to a significant improvement in conversions compared to the control group. This helps in making data-driven decisions in experiments and A/B testing.