Docs / Analysis

Statistical tests

Converise runs a small, fixed set of tests. Every KPI effect names the test it used, or says that none applies.

Which test runs for which KPI

The KPI type picks the test. There is no per-experiment setting, so nobody can choose a method after seeing the data.

KPI type	Test	Needs per variant	Reports
`proportion` (shown as conversion)	Two-proportion z-test, two-sided	Subjects with the event and subjects exposed	p-value and 95% CI of the absolute delta, in percentage points
`quantity` (units per subject)	Welch's t-test, two-sided	Mean, standard deviation, and at least 2 subjects	p-value, degrees of freedom, and 95% CI of the difference of means
`revenue` (money per subject)	Welch's t-test, two-sided	Same as quantity	Same as quantity

Quantity and revenue use the same statistics. They differ only in meaning - units vs money - for the increment estimate and the memo. Relative lift is a point estimate and has no interval of its own.

When the evidence lacks what the test needs, the effect is still shown as a point estimate, marked No inferential statistics in amber, with one of three reasons:

Point estimates only - a conversion KPI came as a rate without counts. Import the per-variant numerator and denominator to enable the z-test.
Variance unavailable - a quantity or revenue KPI has no standard deviation, or a variant has fewer than two subjects. Manual results entry never captures a standard deviation; warehouse bindings, CSV/JSON, and the API do.
KPI type outside the contract - a fallback that current KPI types never hit.

The app claims nothing beyond these two tests and their sequential counterpart (below): no CUPED, no Bayesian output, no one-sided test on experiment evidence. Adding one is an architecture decision, not a setting.

How the numbers are computed

Conversion KPIs. The z statistic uses the pooled standard error; the interval is Newcombe's hybrid score interval, built from the Wilson score interval of each variant, so it keeps its width when a variant has 0 or all conversions. There is no continuity correction. A is the control, B the treatment; x counts subjects with the event, n subjects exposed.

two-proportion z-test

{\hat{p}}_{pooled} = \frac{x_{A} + x_{B}}{n_{A} + n_{B}}

{SE}_{pooled} = \sqrt{{\hat{p}}_{pooled} (1 - {\hat{p}}_{pooled}) (\frac{1}{n_{A}} + \frac{1}{n_{B}})}

z = \frac{{\hat{p}}_{B} - {\hat{p}}_{A}}{{SE}_{pooled}}

p = 2 (1 - Φ (| z |))

l_{A}, u_{A} = \frac{x_{A} + \frac{z^{2}}{2} \mp z \sqrt{\frac{x_{A} (n_{A} - x_{A})}{n_{A}} + \frac{z^{2}}{4}}}{n_{A} + z^{2}}

{CI}_{95%} = [Δ - \sqrt{{({\hat{p}}_{B} - l_{B})}^{2} + {(u_{A} - {\hat{p}}_{A})}^{2}}, Δ + \sqrt{{(u_{B} - {\hat{p}}_{B})}^{2} + {({\hat{p}}_{A} - l_{A})}^{2}}]

l and u are the Wilson score bounds of one variant at z = 1.959963985, Δ = p̂_B − p̂_A. Two identical rates of 10% on 1 000 subjects each give ±2.6 percentage points — equal estimates are not zero uncertainty.

Quantity and revenue KPIs. Welch's test does not assume equal variances, so a variant with a few large orders does not borrow the other variant's spread. x̄ is the per-subject mean, s the standard deviation, ν the Welch-Satterthwaite degrees of freedom.

Welch's t-test

SE = \sqrt{\frac{s_{A}^{2}}{n_{A}} + \frac{s_{B}^{2}}{n_{B}}}

t = \frac{{\bar{x}}_{B} - {\bar{x}}_{A}}{SE}

ν = \frac{{(\frac{s_{A}^{2}}{n_{A}} + \frac{s_{B}^{2}}{n_{B}})}^{2}}{\frac{{(\frac{s_{A}^{2}}{n_{A}})}^{2}}{n_{A} - 1} + \frac{{(\frac{s_{B}^{2}}{n_{B}})}^{2}}{n_{B} - 1}}

p = 2 \cdot P (T_{ν} \geq | t |)

{CI}_{95%} = ({\bar{x}}_{B} - {\bar{x}}_{A}) \pm t_{0.975, ν} \cdot SE

The level is fixed at 95% and every test is two-sided. A configurable level would invite picking one after the fact.
Identical rates give a p-value of exactly 1 and an interval whose width reflects the sample; when both variants of a quantity or revenue KPI have zero variance no test is reported at all (no inferential statistics), because the numbers carry no spread to test a difference against.
p-values and intervals are recomputed on every read from the stored per-variant summaries and are never saved. They show as p < 0.001 below that value, otherwise with three decimals; the rationale quotes four.

Sequential testing: the p-value a decision reads

The classic (fixed-horizon) test is valid only at a horizon chosen before looking. Converise looks at every refresh, so every test-backed result also carries a sequential p-value and interval from the mixture sequential probability ratio test (mSPRT, Johari, Pekelis & Walsh), often called an always-valid p-value. It holds the chance of a false win at 5% however many times the data was looked at, as long as every look is on the cumulative data of the same experiment.

The sequential p is what lets a win be called before the planned end. From the planned end onward the classic test decides - a horizon fixed before anyone looked at the data is exactly the condition under which it is valid. An experiment that never declared an end keeps the stricter bar, because stopping it is always a choice made while looking.
Either way the p is Holm-adjusted across the treatments of the KPI when there is more than one, so an A/B/C test cannot ship on its luckiest arm.
A movement against the KPI is judged on the plain classic p: the safety side stays sensitive, and a false alarm costs a look, not a rollout.
The sequential p is never below the classic p on the same data - roughly 2.5x the traffic for the same call - which is why it applies only where it has to. The expected effect in the workspace review policy (default 10% of the baseline) tunes how quickly a true effect of that size is detected; it never affects validity. Changing it applies to every experiment on its next review.
The KPI table shows one p-value: the one the recommendation reads for that row, with its matching interval in the CI column and the trend chart's band. Hover the value to see both the classic and the sequential p, the one in use marked, and why.

Where the 5% threshold is applied

The tests report p-values and intervals. The threshold α = 0.05 enters only where a number has to become a call, and each place has its own reason.

Place	Rule	Why
Recommendation	`ship` needs p < 0.05 in the good direction on the primary KPI - sequential before the planned end, the classic test from it onward, always Holm-adjusted across treatments - with no guardrail hurt. A significant move the wrong way is `reject` (declined); a non-significant result is `reject` with the reason inconclusive, or precise null when the interval excludes an effect of the expected size. A primary KPI without a test is never called either way.	Naming a direction at p ≥ 0.05 would claim more than the test supports; calling a point estimate a result would claim a test that never ran. Every recommendation carries the verdict, the reason and the exposure action separately.
Confidence	Confidence is about the recommendation, not the effect. A hard fact about the data (a sample ratio mismatch, a source conflict, an established guardrail loss) is high. A call on the effect starts at high and steps down once per warning and per guardrail whose safety is not established. Exploratory segment findings cap at medium; a primary KPI without a test is low and never shipped.	You can be very sure that broken data needs fixing while having no idea about the effect — the two are different questions.
KPI effects table	The chip reads significant, significant decline, or not significant at the same 0.05, on the same p-value as the recommendation (the one in the table's p-value column). A win that is significant only on the classic test before the planned end therefore reads not significant, even where the source tool calls it significant - the tooltip shows both values.	Presentation only. It styles numbers the engine already reported.
Alerts	Significance reached: a primary KPI win at the p that may call it (sequential before the planned end, classic from it onward, Holm-adjusted across treatments), or a decline at classic p < 0.05. Primary KPI regression: the primary moves the wrong way and either p < 0.05 or the relative lift is at least 20% in size.	Both say look now, not decide now. The 20% arm catches a large drop before the sample supports a test.
Segment discovery	The same test runs per segment cell, then Benjamini-Hochberg false-discovery control across all cells. A cell is significant at `q ≤ 0.05`. Cells under 100 subjects on either side are skipped.	Many uncorrected segments produce winners by chance. Findings are exploratory: they can move a recommendation to investigate and cap confidence at medium, never upgrade it to ship.
Cumulative lift chart	Each day carries a 95% band. The chart marks CI excludes 0 since the first day after which every interval excludes zero.	Descriptive. Decisions use the full-window effect, and refresh is daily-only so the chart is not a peeking tool.

"Significant" always means p < 0.05 on the named two-sided test. It does not mean safe to ship: the checks below can still send the result to investigate or iterate.

Checks that are thresholds, not tests

The quality checks never read a p-value or an interval. They look at counts, dates, and the sign of the delta, and a failing check leads to investigate or iterate, never straight to reject.

Check	Threshold	Outcome
Sample ratio mismatch	Chi-square goodness of fit of the observed subject counts against the planned allocation: p < 0.001 is a mismatch, p < 0.01 a drift. Skipped under 200 subjects in total or when a planned arm expects fewer than 5. The deviation in share points is shown as the size of the problem.	A mismatch gives `investigate`, even while running - more data does not fix a broken split, and the cause (a logging gap, a redirect, a filter, an import mapping) is often recoverable, so a rerun is your call, not the engine's.
Insufficient sample size	Any variant under the workspace minimum (500 subjects by default).	`keep_running` while the experiment runs, `iterate` once it is completed — an underpowered result, not an invalid experiment. Harm and data-integrity checks run first: a collapsing guardrail on a small live sample is `investigate`, not `keep_running`. Size the test with the planner below.
Run length vs the plan	The experiment has run fewer calendar days than the workspace minimum (7 days by default - one full weekly cycle; raise it for seasonal traffic), or fewer than it was planned to run without a significant sequential result.	`keep_running` while the experiment runs, `investigate` once it is closed. Inside the plan a win may be called early under the sequential test. An unknown run length (no start date) is never called at all. Risks are never deferred this way - a hurt guardrail still reaches `investigate` first.
Suspiciously short runtime	The daily data spans fewer than 14 days.	Warning. Seven days is the hard floor above; two full weeks is the advisory, because a single covered week can itself be atypical.
Peeking / early stopping	The experiment ended at least one day before its planned end.	investigate unless the win is significant under the sequential test — then the early stop is legitimate and only noted.
Metric conflict and guardrail decline	A guardrail is hurt: it moves against its direction significantly (fixed-horizon p < 0.05), or the interval's worst case exceeds the tolerated loss declared on the KPI (or the workspace default) — a non-inferiority bound.	`investigate` on a hurt guardrail. An adverse move that is not established is safety not established: it lowers confidence and is named, but never declared safe and never blocks. A guardrail within its tolerated loss is safe.
Assignment-unit mismatch	The evidence is aggregated per session while the experiment randomises per user, or similar.	`investigate` with fix the data. A p-value at the wrong grain understates the variance.
Declared segment effects	Each declared cell carries its own test. Under the workspace minimum cell size (100 subjects by default) a segment is low sample size and cannot speak for a decision.	Segment findings are insights: a sign against the global effect, or a win in one segment, changes nothing by itself. Only a reliable cell that is significantly hurt under a global win — surviving Benjamini-Hochberg across the declared cells and with a significant interaction test against the rest of the population — blocks with investigate. Discovery's hurt segments block too (confidence capped at medium); its wins are insights.
Increment estimate	Absolute delta per subject × eligible population × periods × (1 − haircut).	A projection from the point estimate. No interval, no p-value; the haircut is yours to set.

Planning sample size

The runtime planner on the Plan tab uses the standard two-proportion power formula for a conversion KPI. It is an estimate under your assumptions, not a result, and it does not yet cover quantity or revenue KPIs. p₁ is the baseline rate, MDE the relative lift to detect (0.10 = +10%), a the treatment allocation, d the daily eligible subjects.

required subjects per variant

p_{2} = p_{1} (1 + MDE)

n = ⌈ \frac{{(z_{1 - α / 2} + z_{1 - β})}^{2} (p_{1} (1 - p_{1}) + p_{2} (1 - p_{2}))}{{(p_{2} - p_{1})}^{2}} ⌉

days = ⌈ \frac{n}{\min (a, 1 - a) \cdot d} ⌉

Defaults: α = 0.05 two-sided, power = 0.8, allocation 50/50. All three can be edited. The review applies the same α at the end.
Runtime follows the slowest arm: an 80/20 split waits for the 20% variant to reach n.

The free calculator

The significance calculator works on numbers you type, so it offers what the review does not: a one-sided p-value, 90/95/99% confidence, observed power, and a sample ratio check at p < 0.01. None of these apply to a real experiment. The sample size calculator uses the same formula as the planner above.

Reading a result

Check the test name first: a row without one gives direction, not confidence.
Read the interval, not just the chip: +0.4 pp with [0.05, 0.75] is a different case from [0.35, 0.45].
A not significant result on a small sample is inconclusive, not proof of no effect.

PreviousBinding evidence NextImporting experiments