A/B Test

You might say

Do not choose between these sign-up pages by taste. Randomly split users and compare the same metric.

An experiment that randomly assigns eligible users to variants and compares them using predefined metricsEligible new visitors can be randomly shown one of two sign-up pages during the same period, then compared on completion rate. A before-and-after comparison or changing metrics mid-test does not provide the same causal evidence.

Know first

MVP

A/B TestingSplit Test

Randomly assign one visitor pool to A and B for a comparable result

Eligible new visitors

50 / 50

AControl

BVariant

Primary metric: sign-up completionNo decision yetGuardrail: error rate must not increase

What an A/B test needs

Answer one main question: Define the change and primary metric before running the experiment.

Randomize concurrently with matching definitions: Use the same eligible population, event definitions, and time window for both groups.

Predefine the decision and stopping rule: Set primary and guardrail metrics, sample needs, and stopping conditions before launch.

Quick checkChoose the best answer

Which approach is closest to a valid A/B test for two sign-up guides?

During the same period, randomly assign eligible new visitors to A or B, predefine completion and error rates, and analyze after the agreed sample and durationShow B to everyone this month and compare it with A from last monthCheck hourly and stop as soon as one version briefly leads

You can say this to an AI Agent

Draft an A/B test plan for these two sign-up guides. State one hypothesis, eligibility, random assignment, the primary metric, guardrail metrics, analysis window, and stopping rule. Check that both groups use the same tracking definitions. Produce the plan and verification checklist only: do not start the experiment, change traffic, or call a temporary lead a winner. A data specialist should confirm the statistical decision.

Learn next

1Conversion Funnel→2Feature Flag→