A/B test results summary

Experiment: $exp.name

Variant Number of tests Goal: $goal
conversions rate z-score confidence
$variant control ${vdata['t']} no data yet ${vdata['c'][goal]} ${'{0:.1%}'.format(vdata['r'][goal])} ${'{0:.3g}'.format(vdata['z'][goal])} ${'{0:.1%}'.format(vdata['confidence'][goal])}
Insufficient data

For the $goal goal, the $variant variant resulted in ${'{0:.2%}'.format(difference - 1)} more conversions than the $control variant ${'{0:.2%}'.format(difference - 1)} fewer conversions than the $control variant

  • You have now collected enough data to be 99% certain that this conclusion is valid.
  • Recommendation: adopt $variant variant. adopt $control variant.
  • You have collected enough data to be 95% sure that this difference is not simply due to chance factors.
  • Recommendation: let the test run for longer to be completely certain.
  • You don't have enough data to be sure yet that this difference is due to the variant you are testing or purely chance factors.
  • Recommendation: let the test run to collect more data.