Free tool · nothing leaves your browser

A/B test significance calculator

Enter the numbers from both variants. You get a straight answer about whether the difference is real, and every figure behind it explained underneath.

How to use it

1. Variant A is your control — the version that was already there. Put in how many people saw it and how many of them did the thing you care about: bought, signed up, clicked, replied. "Conversions" just means that thing.
2. Variant B is the change you are testing. Same two numbers.
3. Pick your hypothesis. Two-sided is the honest default and is what you want unless you have a specific reason otherwise.
4. Pick a confidence level. 95% is the convention. 99% is a higher bar; 90% is a lower one.

The answer updates as you type. There is no Calculate button because there is nothing to wait for — the arithmetic happens in your browser, in about a millionth of a second.

What each number means

Conversion rate

Conversions divided by visitors. If 570 of 50,000 people bought something, that is 1.14%. This is the only figure on the page that is a plain fact rather than an inference — everything below it is an argument about how much that fact can be trusted.

Relative uplift, and why it is not the same as the gap

1.00% to 1.14% is a rise of 14 percent and a rise of 0.14 percentage points. Both are true and they are not the same number. The relative uplift (14%) is the one worth quoting, because it scales: it says what happens to your revenue. The gap in percentage points (0.14) is the one the statistics work on, and it is what the confidence interval is measured in. Mixing the two up is the single most common mistake in a test write-up, which is why this page labels every one of them.

The p-value

Imagine the change did nothing at all — the two variants are secretly identical and any gap you see is the coin landing heads a few extra times. The p-value is the probability that pure luck would hand you a gap at least this big. A p-value of 0.0157 means: if the change were worthless, you would see a result this good about 1.6 times in a hundred.

It is not the probability that your variant is better. It is not the probability that you are wrong. It is a statement about how surprising your data would be in a world where the change did nothing, and that is all it is. Nearly every misuse of the number comes from forgetting that sentence.

This page reports the one-tailed p-value — the chance of a gap this big in this direction — because that is the figure the widely used calculators show and the one you are most likely to be comparing against. When you choose a two-sided test it also shows the two-tailed p-value, which is simply the one-tailed figure doubled. The verdict above always compares the correct one against the correct threshold, whichever you pick.

Confidence level

The bar you are setting for yourself, decided before you look at the data. 95% confidence means you are willing to be fooled by luck one time in twenty. 99% means one time in a hundred — a higher bar, harder to clear, and it will call a genuine improvement "not significant" more often. The complement of the confidence level (5% at 95%) is called alpha, and "significant" simply means the p-value came in under it.

95% is a convention, not a law of nature. It is not more scientific than 90%; it is more usual. What matters is picking it first and not moving it afterwards because you did not like the answer.

One-sided or two-sided

A two-sided test asks "is B different from A?" and counts a significant drop as a real result. A one-sided test asks only "is B better than A?" and treats any decline as a non-result. One-sided is a lower bar — the same data will more easily be called significant — which is exactly why it is easy to abuse.

Use two-sided unless you genuinely would take no action at all if the new version turned out worse, and you decided that before running the test. In practice almost nobody would ship a change they had just measured as harmful, which means almost everybody should be running the two-sided test. It is the default here for that reason.

Statistical power

Power is about the other mistake. The p-value protects you from calling a difference real when it is not. Power is the chance you would have noticed a real difference of this size, given how much traffic you had. Low power means a test that would probably have missed the effect even if it were genuine — so a "not significant" answer from a low-power test tells you almost nothing. It is not evidence that the change did nothing. It is evidence that you did not run a big enough test to find out.

80% is the usual target. If your result is not significant and your power is far below that, the honest conclusion is "we do not know yet", and the fix is more traffic rather than a different opinion.

The confidence interval

The single most useful number here, and the one most calculators leave out. The p-value tells you whether there is an effect. The interval tells you how big it plausibly is — a range, in percentage points, that the true difference is very likely to sit inside.

An interval of +0.01 to +0.27 percentage points says: the change probably helped, but it might have helped by almost nothing. That is a much more useful sentence than "significant", and it is often the sentence that stops a team from rebuilding a whole checkout flow on the strength of a rounding error.

There is a neat relationship between the two: if the interval does not contain zero, the result is significant, and if it does, it is not. Reporting the interval instead of just the verdict is the easiest single upgrade to how most teams write up a test. The interval shown here is always the two-sided one at your chosen confidence level, even for a one-sided test — a one-sided interval runs to infinity at one end, which is technically correct and reads as broken.

The maths, written out

Nothing here is hidden. The calculator runs a two-proportion z-test, which is the standard test for comparing two conversion rates when both samples are reasonably large.

• Each rate is conversions ÷ visitors, written p₁ and p₂.
• Under the assumption that the change did nothing, both variants share one true rate, so the best estimate of it is every conversion over every visitor: p̄ = (c₁ + c₂) ÷ (n₁ + n₂). That is the pooled rate.
• The pooled standard error is SE = √( p̄(1 − p̄) × (1/n₁ + 1/n₂) ).
• The test statistic is z = (p₂ − p₁) ÷ SE, and the p-value is the area of the normal curve beyond it.
• The confidence interval uses the unpooled standard error — an interval is not computed under the assumption that the change did nothing, so each variant contributes its own variance — giving (p₂ − p₁) ± zcrit × √( p₁(1−p₁)/n₁ + p₂(1−p₂)/n₂ ).
• Power is the chance variant B would clear the threshold that would have been called significant, if its true rate really is the rate you measured.

Because the test and the interval use slightly different standard errors, they can disagree by a hair for a result sitting exactly on the boundary. That is a known property of the two formulas rather than a bug, and it only ever happens where the honest answer was "too close to call" anyway.

The normal distribution is computed here rather than fetched from a library, so the page loads no third-party code and works offline once it has loaded. The whole thing is a few hundred lines and a deploy test checks it against a published calculator's numbers on every push.

Before you trust the number

The arithmetic is the easy part. These are the things that actually make A/B results wrong, and no calculator can catch any of them for you.

Do not stop the test when it turns green

Checking the result every day and stopping the moment it goes significant is called peeking, and it will hand you a "significant" result from pure noise remarkably often. Decide how much traffic the test gets before you start it, then look once. If you must monitor it, treat what you see as a progress bar, not as a verdict.

One test, one question

Test twenty variants at 95% confidence and, on average, one of them looks like a winner for no reason whatsoever. That is what 95% means. If you are comparing many variants or many metrics, you need a higher bar than the one this page applies to a single comparison.

Significant is not the same as worth doing

With enough traffic, a 0.001-point difference becomes significant. Significance says the effect is probably real; the confidence interval says whether it is big enough to bother with. Read them together, always.

Check the split actually split

If your two variants got noticeably different numbers of visitors and they were supposed to be an even split, something is wrong with the test itself — a redirect, a bot, a caching layer — and no amount of statistics will fix a broken assignment.

New is not better

Regular visitors react to anything that changed, and the reaction fades. A week-long test on a site with weekly rhythms can measure the day of the week rather than the change. Run for whole weeks where you can.

Common questions

How many visitors do I need?

It depends entirely on how small an effect you want to be able to see. Smaller effects need dramatically more traffic — halving the effect you want to detect roughly quadruples the sample you need. A practical way to use this page is backwards: put in the traffic you expect and the lift you would be happy with, and look at the power. If the power is low, the test cannot answer your question no matter how it turns out.

My result is not significant. Was the test a failure?

No. It is an answer: you do not have enough evidence. Look at the power and the confidence interval before you decide what to do. High power plus a narrow interval around zero is real evidence that the change did nothing much. Low power plus a wide interval means you learned almost nothing and the traffic was not enough.

Can I use this for something that is not a website?

Yes. Any two groups where each person either did the thing or did not: two subject lines and their open rates, two versions of a form and their completions, two treatments and their success rates. It does not work for averages — revenue per visitor, time on page, score — which need a different test.

Is anything I type sent to a server?

No. The calculation runs entirely in your browser. Nothing is uploaded, nothing is logged, and the page works with your network disconnected once it has loaded.

Is it free?

Yes, with no account, no limit and no catch. It is here because we needed one.