A/B Test Significance Calculator

This A/B test significance calculator tells you whether the difference between two variants is real or likely to be chance. Enter the visitors and conversions for the control and the variant, pick a 90%, 95% or 99% confidence level and a one- or two-sided test, and it shows each conversion rate, the relative lift, the p-value, a confidence interval for the difference and a plain verdict. It also checks for a sample ratio mismatch — a visitor split far from 50/50 that usually signals a tracking bug — and, before you launch, how many visitors per variant you need to detect the lift you care about. Marketers and product teams can paste a CSV of many tests or segments and each is evaluated at once. It is free, runs in your browser, and your experiment data never leaves your device.

Visitors per variant (95% confidence, 80% power)

Baseline rate5% lift10% lift20% lift
1%637,010163,09542,693
2%315,20680,68221,109
5%122,12431,2348,158
10%57,76314,7513,841

When to use it

  1. Decide whether a new landing-page headline really lifted sign-ups.
  2. Check results for mobile and desktop segments separately.
  3. Work out how long to run a checkout test before starting it.
  4. Spot a broken traffic split before trusting a test result.
  5. Evaluate a quarter's worth of email subject-line tests in one go.

How to Use

  1. Enter visitors and conversions for A and B.
  2. Choose the confidence level and one- or two-sided test.
  3. Read the lift, p-value, interval and sample ratio check.
  4. Use the sample size panel before launching, or paste a CSV of tests.

Features

  1. Conversion rates and relative lift
  2. Two-proportion z-test p-value
  3. Confidence interval of the difference
  4. 90, 95 and 99% confidence
  5. One- or two-sided
  6. Sample ratio mismatch check
  7. Sample size per variant
  8. CSV of tests and segments
  9. 100% client-side

FAQs

What does statistically significant mean?

That a difference this large would be unlikely if the variants truly performed the same. At 95% confidence, the p-value must be below 0.05.

What is a p-value?

The probability of seeing a difference at least this big if there were no real difference. Smaller means stronger evidence.

How many visitors do I need?

It depends on your baseline rate and the lift you want to detect. At a 5% baseline, detecting a 10% relative lift needs about 31,000 visitors per variant at 95% confidence and 80% power.

Can I stop the test as soon as it is significant?

No. Checking repeatedly and stopping at the first significant result inflates false positives. Fix the sample size in advance.

Is my data uploaded?

No. Everything is calculated in your browser.

Technical Details

Conversion rates are conversions ÷ visitors for each variant and the lift is (B − A) ÷ A. Significance uses a two-proportion z-test: z = (pB − pA) ÷ √(p̂(1 − p̂)(1/nA + 1/nB)) with the pooled rate p̂, and the p-value from the standard normal distribution, two-sided or one-sided. The confidence interval for the absolute difference uses the unpooled standard error and the critical z for the chosen confidence. A sample ratio mismatch check runs a one-degree-of-freedom chi-square test of the visitor split against 50/50 and warns below p = 0.01. The sample size panel gives visitors per variant for a baseline rate and a relative minimum detectable effect at the chosen confidence and 80% power.

Business & Legal Tools

Business & Legal Tools

  • Quotation Generator
  • Purchase Order Generator
  • Overtime Pay Calculator
  • Final Settlement Calculator
  • Sales Commission Calculator
  • Employee Cost (CTC) Breakdown Calculator
  • Agent Commission Split Calculator
  • LTV and CAC Calculator