What is experiment assignment consistency and why does unit-of-randomization mismatch invalidate A/B test results?

Experiment assignment consistency is the guarantee that a given user (or other randomization unit) sees the same experiment variant on every exposure during the experiment window. Inconsistency — where the same user might see variant A on one visit and variant B on another — introduces assignment noise that inflates the variance of the outcome metric and reduces the effective sample size of the experiment. The unit-of-randomization mismatch is a related but distinct problem: the statistical test assumes that the randomization unit and the analysis unit are the same, because the independence assumption underlying most frequentist statistical tests requires that each observation in the dataset is an independent draw from the population. When a team randomizes at the request level (each HTTP request independently receives a variant assignment) but analyzes at the user level (computing the outcome metric per user, where each user may have seen dozens of requests across both variants), the user-level observations are not independent draws — each user's outcome is a mix of both variants, and the 'treatment effect' for any individual user is the net effect of the mix ratio they happened to receive, not the effect of a consistently-applied treatment. This inflates the variance of the estimated treatment effect, reducing the experiment's statistical power. A 50/50 experiment where assignment is truly random at the request level will, for any given user, produce a binomial distribution of exposures — most users will see both variants, with the majority ratio determined by their total number of exposures. The treatment effect estimated from request-level randomization with user-level analysis is not the effect of being treated, but the effect of a mixed treatment — a signal that exists but is much noisier than the signal from a consistent treatment. The reduction in effective sample size from this mismatch can require 4x to 8x more data to achieve the same statistical power as a correctly-designed experiment with consistent user-level assignment. A user-level assignment requires a stable user identifier (logged-in user ID, deterministic device fingerprint, or durable anonymous identifier stored in a first-party cookie) and a deterministic hash function that maps the user identifier to a variant for a given experiment: hash(user_id + experiment_id) % 100 < treatment_percentage assigns users to treatment consistently across every session without requiring a database lookup or session state.

What is the peeking problem in A/B testing and how does it inflate the false positive rate?

The peeking problem is the practice of checking experiment results before the predetermined sample size is reached and stopping the experiment when the p-value crosses a significance threshold. The standard null hypothesis significance test (NHST) p-value has a validity guarantee that depends on a pre-specified stopping rule: if you commit to checking the result exactly once — after exactly N observations have been collected, where N was calculated before the experiment started — then the p-value correctly controls the Type I error rate at the nominal alpha level (typically 0.05 or 5%). If you check the p-value repeatedly over time and stop when p < 0.05, the validity guarantee no longer holds. The Type I error rate (the probability of concluding there is an effect when there is none) increases with each additional check. For a team that checks results daily for up to 30 days, the actual Type I error rate at the nominal alpha = 0.05 is approximately 20% to 30% depending on the check frequency and the underlying null hypothesis. This means that out of every 10 experiments that genuinely have no effect, 2 to 3 will show a p-value below 0.05 at some point during the 30-day window, and a team that ships winners whenever p < 0.05 will ship 2 to 3 false positives for every 7 to 8 true positives. The aggregate impact is systematic: a product team running 10 experiments per month for a year, shipping winners at p < 0.05 with daily peeking, will ship approximately 30 to 40 features based on false positive results over that year. The customer experience impact of shipping false positives accumulates — each false positive is a feature that consumed engineering time, created product complexity, and did not improve any user metric, while a fraction of the features may actually have harmed metrics in ways that were masked by the noise. The correct solutions are: (1) sequential testing methods (group sequential tests, alpha-spending functions, always-valid inference) that maintain the Type I error rate guarantee under continuous monitoring by adjusting the significance threshold at each look — the threshold starts lower than 0.05 at early looks and converges toward 0.05 as the sample size approaches the pre-specified maximum; (2) Bayesian experimentation, which does not use p-values and does not have a peeking problem in the same form, but requires a prior distribution specification and a different interpretation of results; or (3) strict pre-commitment to a fixed sample size and a single look, with organizational enforcement against early stopping.

What is the difference between mutual exclusion groups and layered experiments in A/B testing infrastructure?

Mutual exclusion groups and layered experiments are two models for managing concurrent experiments that address the problem of experiment interaction effects — where two experiments running simultaneously on the same user population affect each other's estimated treatment effects. In a mutual exclusion model, each user is placed in at most one experiment at a time within a group. The traffic is partitioned: if experiment A uses 10% of traffic and experiment B uses 10% of traffic, those 10% slices are drawn from non-overlapping portions of the user population — a user in experiment A's treatment cannot also be in experiment B's treatment. Mutual exclusion guarantees that no user sees the treatment of more than one experiment simultaneously, eliminating interaction effects entirely. The cost is throughput: if 50 experiments are running simultaneously at 10% traffic each, mutual exclusion requires 500% of traffic — more traffic than exists. In practice, mutual exclusion models have a maximum concurrent experiment throughput determined by the total traffic volume and the minimum per-experiment sample size requirement. A mutual exclusion group with 100% of traffic and 10 experiments requires each experiment to use at most 10% of traffic, reducing to 5% if 20 experiments must run simultaneously. A layered experiment model assigns users independently for each experiment, using a different hash space for each layer. A user's assignment in layer A is determined by hash(user_id + 'layer_A' + experiment_id_A) and their assignment in layer B is determined by hash(user_id + 'layer_B' + experiment_id_B). These two hashes are independent — knowing a user's layer A assignment tells you nothing about their layer B assignment — so users can be in both experiments simultaneously. The layered model allows an unlimited number of concurrent experiments (in principle) and is used by large platforms (Google, Facebook, Microsoft) that run thousands of simultaneous experiments. The validity assumption is that the experiments are orthogonal: the treatment effect of experiment A is the same regardless of which variant of experiment B the user is in. This assumption holds when experiments target different product surfaces or different user behaviors, but may not hold when two experiments modify the same user experience in ways that interact — for example, an experiment on the color of a button and an experiment on the text of the same button may interact because users respond to the combination of color and text differently than to either change in isolation. Layered experiments do not guarantee the absence of interaction effects; they assume their absence and require explicit validation when interactions are suspected.

How does the minimum detectable effect determine the minimum rollout granularity for an A/B experiment?

The minimum detectable effect (MDE) is the smallest treatment effect that the experiment is powered to detect with the specified statistical power (typically 80% or 90%) and significance level (typically alpha = 0.05). The MDE, combined with the baseline metric value, the variance of the metric, the traffic rate, and the experiment duration, determines the required sample size: the number of users (or other randomization units) that must be observed in each variant for the experiment to have sufficient statistical power to detect the MDE. The required sample size, divided by the daily traffic rate, gives the minimum run duration for a 100% traffic allocation. The minimum rollout granularity — the smallest fraction of traffic that the team can validly use for an experiment — is determined by the relationship between the required sample size and the acceptable run duration. If the required sample size for a 10% MDE at 80% power is 80,000 users per variant (160,000 total), and the product has 10,000 daily active users, the minimum run duration at 100% traffic allocation is 16 days. If the team wants to limit experiment exposure to 10% of traffic to reduce risk, the run duration becomes 160 days — too long to be operationally useful. If the team wants to limit run duration to 14 days, the minimum traffic allocation is approximately 57% (80,000 / (10,000 × 14) per variant). The 10% MDE at 14-day run duration requires approximately 57% traffic allocation; smaller traffic allocations cannot achieve 80% power for a 10% MDE within 14 days. The rollout granularity is not a product decision about risk tolerance; it is a consequence of the traffic rate, the metric variance, and the MDE. A team that wants to run experiments at 1% traffic allocation to reduce blast radius must either accept much longer run durations, accept lower statistical power (meaning more false negatives — real effects that are not detected), or target larger MDEs (meaning only large effects are detectable). These tradeoffs must be documented in the A/B testing infrastructure decision record so that the product team understands what experiment designs are valid given the infrastructure constraints, rather than discovering the constraints by running experiments that were underpowered from the beginning.

The A/B testing infrastructure decision record: why the experimentation platform you chose determines your statistical validity ceiling and your rollout granularity

July 10, 2026 · WhyChose

A/B testing infrastructure is set up once, during a sprint that is framed as "we should start A/B testing," and then it runs for years without the decisions behind it being revisited as decisions. The team picks an assignment model — perhaps a server-side feature flag, perhaps a client-side random call, perhaps a third-party experimentation platform — and moves on to running experiments. The assignment model, the statistical framework, the concurrent experiment model, and the metric hierarchy are all treated as implementation details rather than architectural decisions with long-term consequences. They determine what the team can validly learn from experiments for as long as the infrastructure is in use, and changing them requires invalidating all in-flight experiments and retraining everyone who interprets results. The decisions are effectively permanent, but they are never written down as decisions.

The missing decision record is invisible in success. When experiments produce clean, significant results that match product intuition, the statistical framework appears to be working correctly. A new onboarding flow shows p = 0.03, the team ships it, and activation rate goes up — the framework validated itself. The invalidity of the framework surfaces in disagreement or in long-term review: a result that contradicts intuition is scrutinized, the analyst re-examines the assignment model, and discovers that the randomization unit was the HTTP request rather than the user; or a quarterly review of shipped experiments finds that the holdout group has identical metrics to the treatment group despite 30 features having been shipped based on statistically significant results. By the time the framework flaw is identified, months or years of experiment results may be unreliable, and the team cannot determine retroactively which results were valid and which were artifacts of the infrastructure's statistical properties.

The rollout granularity — the minimum fraction of traffic that can be used for a valid experiment — is not a product decision about risk appetite; it is a consequence of the infrastructure decisions. The minimum valid rollout fraction is determined by the relationship between the minimum detectable effect the team wants to measure, the metric variance, the traffic rate, and the acceptable experiment duration. A team that wants to test changes on 0.5% of traffic to limit blast radius must have sufficient traffic to collect the required sample size in an acceptable time window, and must have an assignment infrastructure that can consistently route 0.5% of users to a variant over a multi-week period. A team whose daily active user count and metric variance require 60,000 users per variant to detect a 10% improvement at 80% power cannot run valid experiments at 0.5% traffic allocation without a run duration of over a year. These constraints are derivable from first principles before the first experiment is designed, but only if someone does the derivation before the infrastructure is built — which requires treating the experimentation platform as an architectural decision rather than a tactical implementation.

Two things that happen when the decision is not written down

The unit-of-randomization mismatch: an 8-week test that failed to detect a real 14% improvement

A 40-person B2B SaaS company built project management software for small engineering teams. Their product had a task detail panel — the side drawer that appeared when a user clicked on a task — that the design team had redesigned substantially: the new version reorganized the fields, surfaced the dependency graph inline rather than in a separate tab, and moved the status controls to a more prominent position. The engineering team was confident the redesign would improve task completion rates. They wanted to validate before rolling it out to all 14,000 active users and set up an A/B test.

The frontend was a React single-page application. The engineer implementing the experiment added an assignment call at the component level: when the task detail panel rendered, it called a function that returned a variant based on Math.random() < 0.5. If the result was true, the new panel design rendered; if false, the original design. The engineer noted in the pull request that users might see both designs within a session but assumed this would "even out" statistically. The assignment was not persisted between renders or between sessions — each panel open was an independent coin flip.

The experiment ran for 8 weeks. The primary metric was task completion rate: the fraction of tasks that were marked "done" within 7 days of their creation date. The analyst computed the metric by querying the task table, grouping by experiment_variant (a field that logged the variant seen on the last task detail panel open for each task), and computing the done-within-7-days rate for each group. The result: control = 61.2% completion, treatment = 62.7% completion. The difference was 1.5 percentage points. The p-value from a two-sample z-test for proportions was 0.41 — far from the 0.05 threshold. The team concluded that the redesigned panel had no measurable effect on task completion and reverted the feature branch. The designer was disappointed. The PM noted the result in the product decision log as "redesign tested, no effect detected."

What actually happened: the Math.random() call on each render meant that a user who opened the task detail panel 20 times in 8 weeks would see the new design for roughly 10 of those opens and the original design for the other 10. The assignment was independent across renders — not across users, sessions, or even individual tasks. The experiment_variant field on each task recorded the variant from the most recent panel open, which might be different from the variant that was showing when the user actually interacted with the task and completed it. A user who completed a task 5 minutes after an open in the original variant would have their completion attributed to the original variant even if their prior 15 panel opens on that task were in the new variant.

The core problem was unit-of-randomization mismatch: the randomization unit was the panel render (each render was an independent trial), but the analysis unit was the task (a task has a single outcome — completed or not — aggregated over all panel opens). The independence assumption underlying the z-test for proportions requires that each task's outcome is an independent observation from one treatment group. When the task's effective treatment is a random mix of both variants, this assumption fails. The outcome variance per task is inflated by the noise in the assignment mix: two users with identical baseline completion behavior will show different completion rates in the dataset not because of the treatment effect but because their random assignment mix happened to differ. This inflated variance requires a much larger sample size to detect the true treatment effect with the same statistical power.

A back-of-envelope calculation: the task completion rate had a baseline of approximately 61% and a standard deviation across tasks of roughly 0.49 (the standard deviation of a Bernoulli random variable with p=0.61). A correctly-designed experiment targeting a 10% relative improvement (61% → 67%) with two-sided alpha=0.05 and 80% power would require approximately 1,200 tasks per variant under consistent user-level assignment. The experiment had 14,000 active users and an average task creation rate of 3.2 tasks per user per week, giving approximately 45,000 tasks per week across the platform. At 50% traffic, 22,500 tasks per week per variant — the 1,200-task requirement should have been met in less than a week.

But the actual effect observed in a retroactive analysis (comparing users whose average assignment, across all their panel opens, was above 70% treatment to users below 30% treatment) was a 14% relative improvement in completion rate — 61.2% vs. 69.7%. The mixed-assignment noise had so thoroughly diluted the signal that 8 weeks of data and 180,000 tasks were insufficient to detect a 14% real effect. The effective sample size for the original experiment, accounting for the assignment inconsistency and unit mismatch, was equivalent to approximately 300 tasks per variant — 4x less than the minimum required for 80% power at a 10% MDE. The designer's redesign genuinely improved task completion by 14%. It was never shipped because the experiment infrastructure used for the test was invalid for the analysis that was performed on it.

The peeking problem and the novelty effect: 18 months of false positive wins and a flat holdout metric

A 75-person consumer fintech company ran a personal finance platform. Their product analytics showed that users who invested at least once per month had 5x the 12-month retention of users who did not, making investment engagement the north star metric for the product team. Over 18 months, the product team ran 47 A/B experiments designed to increase investment deposit behavior: new dashboard widgets, push notification schedules, onboarding flows, email cadences, educational content placements, and UI redesigns of the investment section. They shipped 24 winners — experiments that showed a statistically significant increase in investment deposits at p < 0.05.

The experimentation infrastructure used session-based assignment (a session cookie set on first visit, reset on browser clear) and a JavaScript analytics library that computed p-values for each experiment on demand. The product team's practice was to check experiment results daily in the analytics dashboard. If a result crossed p < 0.05 on any day, the PM responsible for that experiment could call it a win and ship. If an experiment ran for 4 weeks without crossing the threshold, it was killed as "no effect." The practice was adopted from a blog post about "moving fast with data" that the team's head of growth had circulated during an all-hands. No one had computed a sample size requirement for any experiment. No one had specified a minimum detectable effect. No one had set a minimum run duration. The threshold was "significant before 4 weeks means ship it."

The investment recommendation widget was the most prominent shipped winner. The experiment ran from week one of the program. Users in the treatment group saw a new widget on the dashboard home screen that displayed a personalized investment recommendation — "Based on your cash balance, you could invest $200 this month and reach your emergency fund target in 4 months." The metric was 30-day investment deposit amount. On day 6, the p-value crossed 0.047. The treatment group's 30-day investment deposits were 22% higher than the control group's. The PM called it a win. Engineering spent 3 weeks hardening the recommendation engine for full rollout. The widget shipped to all users in week 7 of the program.

By month 15, the product team had become concerned. The investment engagement rate — the fraction of monthly active users who made at least one investment deposit in a given month — had been flat at 18% for the past 12 months despite 24 shipped improvements. The analytics dashboard showed healthy p-values on shipped experiments; the overall engagement metric showed no trend. The CTO asked the data team to investigate. The data team ran a holdout experiment: 10% of new users were randomly assigned to a holdout group that was excluded from all subsequent experiments. The holdout group and the treatment group would be compared after 90 days.

The holdout result, at 90 days: investment engagement rate in the holdout group = 18.3%. Investment engagement rate in the full treatment group (users who had received the benefit of all 24 shipped improvements) = 18.1%. The 24 shipped improvements had produced a cumulative effect of negative 0.2 percentage points — statistically indistinguishable from zero and directionally negative. The 22% improvement in the investment recommendation widget experiment had not sustained: new users who saw the widget for the first time invested at a higher rate in their first week (the novelty effect — new features attract attention and trial behavior regardless of long-term utility), and the experiment had been stopped on day 6, when the novelty peak was at its maximum. The sustained effect was zero.

The peeking problem compounded the novelty effect. The standard frequentist p-value at alpha=0.05 is calibrated for a single look at the data after the pre-specified sample size is collected. For the investment widget experiment, the analyst checked results daily for 6 days before stopping. Simulation of this peeking pattern — daily checks, stop at p < 0.05 or 30 days, both null and alternative hypothesis scenarios — estimates the actual Type I error rate for this procedure at approximately 26%. Not 5%. For the 47 experiments run in 18 months, approximately 10 to 12 of the 23 experiments that did not cross the threshold (killed as "no effect") likely had real positive effects that the test failed to detect because it was stopped early. Approximately 6 to 8 of the 24 shipped winners were false positives — real investment deposits in the first week, driven by novelty, with no lasting effect. The product team had shipped 6 to 8 false positives and killed 10 to 12 real improvements in 18 months, with investment engagement flat the entire time. The experimentation program had negative expected value for the company.

The data team's post-mortem identified four infrastructure deficiencies that had enabled the false positive accumulation: no pre-specified sample size or MDE; no minimum run duration to allow the novelty effect to decay (2 to 3 weeks is the standard washout period for novelty-driven behavior change); no sequential testing procedure that maintained validity under daily peeking; and no holdout group established at the beginning of the program to detect aggregate null effects. All four deficiencies were infrastructure decisions that had never been documented as decisions — they were the absence of decisions, which is itself a decision with consequences that accumulated over 18 months before they became visible.

Three structural properties that are set at experimentation platform selection time

The assignment layer and the randomization unit ceiling

The assignment layer is the component of the system that receives a user identifier and an experiment specification and returns a variant assignment. The layer can be the CDN edge (assignment happens at the CDN before the origin server receives the request), the API gateway (assignment happens at the reverse proxy before the application code runs), the application backend (assignment happens in the service that handles the request), a dedicated feature flag service (a separate microservice that the application calls to resolve assignments), or the client (the browser or mobile app resolves assignment locally using a downloaded set of experiment configurations). Each choice has distinct tradeoffs on latency, consistency, targeting capability, and the ability to use server-side state for assignment decisions.

The randomization unit is the entity over which assignment is consistent. The highest-value randomization unit for most product experiments is the user — a logged-in user always sees the same variant, across devices, sessions, and time. User-level randomization requires a stable user identifier at the point of assignment. If the assignment layer is the CDN edge and the request is anonymous (no authentication cookie), the CDN cannot assign by user ID because it does not know the user. Edge assignment for logged-in users is possible only if the authentication cookie or JWT is available at the edge and can be decoded to extract the user identifier. If the product has a significant anonymous traffic fraction (pre-registration flows, landing pages, unauthenticated API endpoints), user-level assignment at the edge requires routing authenticated and anonymous traffic through different assignment paths. Session-level and device-level randomization are lower consistency units: a user who clears their cookies or switches devices will receive a new assignment, potentially becoming a member of both variant groups across their lifetime. For experiments where the outcome metric is measured per-session rather than per-user, session-level randomization is valid. For experiments where the outcome metric requires multi-session measurement (30-day retention, monthly deposit behavior), session-level randomization produces the same unit mismatch problem as the render-level assignment in the task panel example — the observed user is a blend of variants across sessions, not a consistent recipient of one variant.

The targeting attributes available at assignment time are bounded by the assignment layer's position in the request path. A CDN edge cannot access the user's subscription tier, purchase history, or cohort membership without a cache lookup or a call to the application backend — both of which add latency and architectural complexity to what is otherwise a zero-latency assignment. An application backend can access any attribute available in the service's database or in-memory cache. A dedicated feature flag service can access any attribute passed to it at call time, making the targeting surface bounded by what the calling service decides to include in the assignment request. The targeting capability ceiling is set at assignment layer selection time: a team that needs to run experiments targeting users with specific behavioral or demographic attributes must choose an assignment layer that can access those attributes, either by position in the request path or by parameter passing. This connects to the authentication strategy decision record: the authentication model determines which user attributes are available in the authenticated session context, and those attributes determine the targeting surface for assignment-time conditions. A JWT that encodes subscription tier and user segment allows the API gateway to do segment-targeted assignment without a backend call; a session cookie that contains only a session ID requires a cache lookup to resolve user attributes for targeting.

The assignment model must also specify the hash function and the salt strategy. Deterministic assignment — where the same user ID in the same experiment always receives the same variant — is implemented by hashing the combination of user identifier and experiment identifier: murmur3(user_id + ':' + experiment_id) % 100 assigns users to percentage buckets consistently without storing any assignment state. The experiment identifier acts as a salt, ensuring that the same user is independently assigned across different experiments (user A in experiment X being in variant A tells you nothing about whether user A is in variant A or B in experiment Y). The hash function must be computationally cheap (murmur3, xxhash, or a simple CRC variant — not SHA-256, which is 10 to 50x slower and unnecessary for non-cryptographic use), non-correlated across different experiment IDs, and consistent across languages (if the assignment is computed in both the backend and the client for different use cases, the same hash function implementation must be used in both). This connects to the caching strategy decision record: if experiment assignments are computed at request time using a deterministic hash, no caching is required — the same computation produces the same result. If assignments are resolved by calling a feature flag service, caching the service's response in the calling service (with an appropriate TTL and cache invalidation strategy for mid-experiment configuration changes) reduces latency and reduces load on the flag service, but requires defining what happens when a cached assignment becomes stale: a user who is moved between variants by a mid-experiment reconfiguration should the cache expose them to both variants during the TTL window?

The statistical framework and the validity ceiling

The statistical framework specifies what the experiment results mean and what error rate the team is operating at. A framework that is not specified produces whatever defaults the analytics tool uses — which may or may not match the team's actual operating conditions. The validity ceiling of the framework is the maximum confidence the team can have in any single experiment result, given the framework's properties. A framework with daily peeking and a 30% Type I error rate has a validity ceiling of 70%: in the best case, 70% of significant results are true positives. A framework with pre-specified sample sizes, a single look, and alpha=0.05 has a validity ceiling of 95% — 95% of significant results are true positives (assuming all other assumptions hold: the metric is the right metric, the assignment is consistent, the population is stationary over the experiment window). The validity ceiling is set at framework selection time, because changing the framework mid-program requires discarding all in-flight experiments and either retroactively invalidating historical results or documenting that results before a given date were produced under a different framework with a different validity ceiling.

The minimum detectable effect (MDE) is the smallest treatment effect the framework is designed to detect at the specified power level. The MDE determines the required sample size: a smaller MDE (detecting a 2% improvement) requires a much larger sample size than a larger MDE (detecting a 20% improvement) for the same power level and metric variance. The sample size formula for a two-sided proportions test at power 1-β and alpha α with baseline rate p and MDE Δ is approximately n = 2 × (z_α/2 + z_β)² × p(1-p) / Δ² per variant. At p=0.6, alpha=0.05, power=0.8, and Δ=0.06 (10% relative MDE on a 60% baseline), the required sample size is approximately 1,680 per variant. At Δ=0.012 (2% relative MDE), the required sample size is approximately 42,000 per variant — 25x more data for a 5x smaller MDE. The MDE is a product decision (what is the smallest improvement worth detecting?) but it translates directly into an infrastructure constraint (how much traffic is required and for how long?). This calculation must be done before the first experiment is run, because the traffic rate and the metric baseline determine whether the desired MDE is achievable within an acceptable run duration at any traffic allocation.

The novelty effect is a temporal confound that is independent of the statistical framework: new features attract curiosity and trial behavior from users who encounter them for the first time, which can produce a short-term metric improvement that decays to zero as the novelty wears off. The novelty effect confound requires a minimum run duration policy — the experiment must run long enough for the novelty-driven behavior to decay and the steady-state treatment effect to dominate. The minimum run duration for novelty washout depends on the product's engagement cadence: a daily-use productivity tool may reach steady-state within 7 days; a monthly-use financial tool may require 4 to 6 weeks for the novelty effect to decay. The minimum run duration policy must be specified in the A/B testing infrastructure decision record and enforced by the analytics tooling — the experiment should not be stoppable early except for safety reasons (a strong harm signal on a guardrail metric), and the analytics dashboard should not display actionable p-values until the minimum run duration has elapsed. If the dashboard displays live p-values from day one, the peeking pressure is structural: teams will look at the data, and if the p-value looks significant at day 6, stopping the experiment becomes difficult to resist regardless of the policy. This connects to the feature flag decision record: the feature flag infrastructure and the A/B testing infrastructure are often built separately and then integrated, but the integration point — the moment when a winning variant is "shipped" to 100% of traffic — must be explicitly designed. A feature flag that allows instant 100% rollout creates the same peeking pressure as an analytics dashboard: if the engineer can ship to 100% at any time, the temptation to stop the experiment early when results look good is structural. The decision record should specify who can change the traffic allocation of an active experiment, under what conditions, and what the review gate is before a winner is declared and rolled out.

Sequential testing methods maintain the Type I error rate guarantee under continuous monitoring by using an adjusted significance threshold at each look. Group sequential tests (Pocock boundaries, O'Brien-Fleming boundaries, Lan-DeMets alpha-spending) specify the sequence of significance thresholds for planned interim looks: an experiment with 4 planned looks (weeks 1, 2, 3, 4) using an O'Brien-Fleming boundary would use adjusted thresholds of approximately 0.0005, 0.014, 0.045, 0.050 at each look, compared to the fixed 0.050 threshold of a single-look test. The thresholds are chosen so that the probability of falsely rejecting the null at any look is capped at the nominal alpha=0.05 across the full test. Always-valid inference (mSPRT, anytime-p-values) is a related approach that allows continuous monitoring with a validity guarantee at every point in time — the p-value can be checked at any point after experiment start and the Type I error guarantee holds. The tradeoff is power: always-valid inference requires more data than a fixed-sample test to achieve the same power at the pre-specified sample size, because the validity guarantee at all intermediate points constrains the test. The framework selection is a tradeoff between monitoring flexibility and statistical efficiency, and it must match the team's actual behavior: a team that will check results daily should use a sequential method that accounts for daily checks; a team that commits to a single look after 4 weeks can use the more efficient fixed-sample test.

The concurrent experiment model and the interaction validity ceiling

The concurrent experiment model determines how multiple experiments running simultaneously on the same user population interact. The interaction validity ceiling is the maximum number of simultaneous experiments for which the results are guaranteed to be uncontaminated by interaction effects. For a mutual exclusion model, the interaction validity ceiling is the number of experiments that can fit into the total traffic without overlap — at 10% traffic per experiment, the ceiling is 10 simultaneous experiments at 100% total traffic allocation. For a layered model, the interaction validity ceiling is theoretically unlimited, but the independence assumption that the model relies on must be explicitly verified for each pair of concurrent experiments that might interact.

The interaction detection requirement is the operational mechanism for maintaining the interaction validity ceiling in a layered model. A product team running 20 simultaneous experiments on a shared user population must have a process for identifying experiment pairs that violate the independence assumption before those experiments are launched together. The process can be rule-based (no two experiments may modify the same user-facing element), review-based (all new experiments must be reviewed against the current experiment list for potential interactions), or empirical (the analytics system computes interaction effects for all pairs of concurrent experiments by testing whether the treatment effect in experiment A differs significantly between the treatment and control groups of experiment B). The empirical approach is the most reliable but the most computationally expensive: detecting pairwise interactions in a set of 20 concurrent experiments requires 190 pairwise comparisons, each requiring enough data to detect a realistic interaction effect size. For teams running many simultaneous experiments with large traffic volumes, the empirical approach is standard (Google's experimentation infrastructure checks for interactions automatically). For smaller teams, a rule-based or review-based approach is more practical, with empirical checking reserved for experiment pairs that the rule or review process flagged as potentially interactive.

The traffic allocation model — how traffic is divided among concurrent experiments — interacts with the concurrent experiment model in ways that must be documented explicitly. In a mutual exclusion model, the traffic allocation must sum to 100% or less across all concurrent experiments in the same exclusion group, or some users will be excluded from all experiments (which wastes traffic that could have been used for learning) or the model's constraint will be violated. In a layered model, each layer's traffic allocation is independent of other layers, but the joint probability of a user being in multiple experiment treatments simultaneously is the product of the individual allocations — a user has a 5% chance of being in the treatment group of experiment A and a 5% chance of being in the treatment group of experiment B, giving a 0.25% chance of being in both treatment groups simultaneously. For detecting interaction effects, the fraction of users in both treatments simultaneously must be large enough to collect the required sample size — a 0.25% simultaneous treatment fraction requires a very large total user population or a very long run duration to accumulate enough interaction-group data for reliable detection. This connects to the microservices vs. monolith decision record: a microservice architecture with independent deployment of services creates a natural boundary for experiment layering — service A can run experiments independently of service B if the services do not share the user experience surface being measured. A monolith that renders the full user interface cannot layer experiments at the service level because all experiments affect the same rendering pipeline; the concurrency model for monolith experiments is typically mutual exclusion within the rendering layer. Teams that migrate from monolith to microservices often find that the experiment concurrency model must be redesigned alongside the architecture — the mutual exclusion constraints that were manageable with one rendering surface become a throughput bottleneck when the same user-facing team is now responsible for experiments across five microservices whose results they want to run simultaneously.

The experiment interaction model must also specify what happens to in-flight experiments when a new experiment is launched that might interact with existing ones. In a mutual exclusion model, launching a new experiment that shares traffic with existing experiments requires resizing the existing experiments' traffic allocations — which invalidates the existing experiments' data, because the user population in the experiment changes mid-flight. A user who was in control group of experiment A at 20% traffic may be moved out of experiment A's population when the traffic is reduced to 15% to make room for a new experiment. The resizing policy — whether mid-flight resizing is allowed, what the minimum resizing increment is, and what the analyst must do with data collected before and after the resize — must be documented. Many teams discover the resizing problem only when they need to launch a high-priority experiment and find that existing experiments are consuming all available traffic, and the decision to resize or delay must be made under pressure rather than from a documented policy. The policy belongs in the A/B testing infrastructure decision record because it determines the team's ability to respond to urgent product decisions while maintaining the validity of ongoing experiments.

Five sections the A/B testing infrastructure decision record should address

1. Assignment model and randomization unit selection

Document the assignment layer (edge, gateway, backend service, feature flag service, or client), the randomization unit (user ID, session ID, device ID, account ID, or request), and the hash function and salt strategy. For each choice, document the rationale: why was user-level assignment chosen over session-level? What user identifier is used, how is it generated, and what is its stability guarantee (a UUID assigned at registration versus a session cookie that resets on browser clear)? What is the targeting attribute surface at the assignment layer — which user attributes are accessible at assignment time without an additional database query, and which require a backend call? What is the assignment latency at the 99th percentile, and is that latency acceptable for the product's performance requirements?

Document the behavior under edge cases: what happens when the user identifier is unavailable (anonymous users, unauthenticated API calls)? Is the user excluded from the experiment, assigned to control, or assigned using a fallback identifier (session ID, IP address range)? What happens when a user is authenticated mid-session — does their assignment change from the anonymous fallback to the user-level assignment? If so, is there a risk of the same user appearing in both the control and treatment groups (once as anonymous, once as authenticated) in the analysis? The holdout for exclusion from experiments — what fraction of users is permanently excluded from all experiments to serve as an absolute holdout baseline — should also be documented here, including whether the holdout is enforced at the assignment layer or implemented by a post-hoc analysis. A permanent holdout of 5% to 10% of users provides a long-term baseline for detecting aggregate effects of the experiment program, and the decision to maintain a permanent holdout must be made at infrastructure setup time because it requires allocating traffic that is never available for experiments. This connects to the API versioning decision record: if the A/B testing infrastructure exposes an assignment API (a service that other services call to resolve variant assignments), that API's versioning and compatibility guarantees affect every service that depends on it. A breaking change to the assignment API — changing the return format, removing a field, or changing the hash function — invalidates in-flight experiments in all dependent services simultaneously.

The analysis unit must be documented explicitly alongside the randomization unit, and the two must match. If randomization is by user ID, the analysis must aggregate metrics at the user level before computing treatment effects — a task completion rate experiment must count the number of tasks completed per user and compare the distributions between variants, not count the total tasks completed and divide by the total tasks observed (which conflates user-level effects with user-volume effects). The analyst's documentation of the analysis procedure — the SQL query or the notebook that computes the primary metric — should be reviewed against the randomization unit at experiment design time to catch unit mismatches before the experiment runs rather than after it fails to detect a real effect.

2. Statistical framework and stopping rule policy

Document the statistical framework (frequentist NHST, sequential testing, Bayesian), the significance level (alpha), the power target (1 - beta), the default minimum detectable effect for the primary metric category (conversion rate, engagement metric, revenue metric), and the stopping rule (fixed sample with single look, group sequential with planned interim looks, or always-valid inference with continuous monitoring). The stopping rule policy is not optional documentation — it is the mechanism that maintains the Type I error guarantee. An analytics platform that displays p-values from experiment start implicitly allows continuous monitoring regardless of the documented policy; the platform must enforce the stopping rule by withholding or explicitly flagging results until the minimum run duration and minimum sample size have been reached.

Document the sample size calculation for the most common experiment types, using the actual baseline metric values and variances from the product's historical data. For a conversion rate experiment on a metric with a 15% baseline conversion rate and a target MDE of 3 percentage points (a 20% relative improvement), at alpha=0.05 and power=0.80, the required sample size is approximately 1,400 per variant (2,800 total). At the product's current daily active user count and 50% traffic allocation, what is the minimum run duration to reach this sample size? If the DAU is 5,000, the run duration at 50% traffic and 50% conversion to the experiment surface is 2,800 / (5,000 × 0.5) = 1.12 days. If the DAU is 500, the run duration is 11.2 days — plus the minimum novelty washout period of 7 to 14 days, making the minimum run duration for a valid experiment approximately 18 to 25 days. These calculations should be pre-computed for the most common experiment types and documented as the experiment design constraints that PMs and engineers use when proposing experiments. A PM who wants to run an experiment and get results in 3 days needs to know that the minimum valid run duration is 18 days; this sets the expectation before the experiment is launched rather than triggering a post-mortem debate about whether the 3-day result can be trusted. This connects to the CI/CD pipeline decision record: the deployment pipeline must be able to maintain a consistent experiment configuration throughout the run duration without an accidental deployment changing the assignment logic mid-experiment. A deployment that changes the feature flag service configuration or the assignment hash logic mid-experiment invalidates the data collected before the change.

Document the multiple testing correction policy for teams that run concurrent experiments that share a common primary metric. If 15 experiments are running simultaneously and each targets a 5% false positive rate independently, the probability that at least one experiment shows a false positive is approximately 54% (1 - 0.95^15). If the team is willing to accept a 5% family-wise error rate — a 5% chance that any experiment in the concurrent set is a false positive — the per-experiment alpha must be adjusted using Bonferroni (divide alpha by the number of concurrent experiments) or Benjamini-Hochberg (control the false discovery rate). The multiple testing correction policy must specify which sets of concurrent experiments require correction: all experiments running at the same time, only experiments targeting the same primary metric, or experiments targeting the same user segment. The policy must also specify the correction procedure to use and how it is applied — if a team is using always-valid inference, the sequential validity guarantee is per-experiment, not for the family of concurrent experiments, and a separate multiple testing correction is required for family-wise control.

3. Concurrent experiment model and interaction management

Document the concurrency model (mutual exclusion, layered, or hybrid), the maximum concurrent experiment count at each traffic allocation, and the traffic resizing policy for in-flight experiments. For a layered model, document the independence assumption and the process for validating it: which product surfaces or user behavior dimensions are considered independent by default, which pairs of surfaces require an explicit independence review before simultaneous launch, and which pairs are always placed in mutual exclusion (because the interaction effect is known or likely). The independence assumption is most likely to be violated when two experiments modify adjacent elements in the user experience — the same page, the same flow, or the same decision point — or when two experiments both aim to change the same user behavior (two experiments both trying to increase investment deposits will interact because any improvement from experiment A changes the baseline that experiment B is measuring).

Document the interaction detection mechanism: how does the team discover that two concurrent experiments are interacting before the results are analyzed? A pre-launch review process — where the experiment owner checks the current experiment list for potential interactions and notes any pairs that might interact — is the minimum viable mechanism. An automated check that flags any new experiment targeting a surface within N user actions of an existing experiment's surface is more systematic and less dependent on the experiment owner's knowledge of all concurrent experiments. The audit trail for interaction reviews should be part of the experiment record — each experiment's documentation should note which concurrent experiments were reviewed for interactions and what the outcome of the review was. Without the audit trail, a post-mortem on a suspicious experiment result cannot determine whether an interaction review was done or skipped. This connects to the observability platform decision record: the observability platform must be able to segment experiment metrics by the variant assignments of all other concurrent experiments to support interaction detection. A metric dashboard that shows the primary metric trend for experiment A, broken down by whether the user was in the treatment or control group of experiment B, allows the analyst to detect interaction effects after the fact. Configuring this segmentation for all pairs of concurrent experiments is computationally expensive at scale, but for teams running fewer than 20 concurrent experiments, it is feasible with a standard columnar analytics database (BigQuery, Snowflake, Redshift).

For the mutual exclusion model, document the exclusion group definitions and the governance process for adding new groups. An exclusion group is a set of experiments that cannot run concurrently because they share a traffic pool. The group definitions must be specific: which experiments belong to each group, what traffic percentage is allocated to each group, and who is authorized to add an experiment to a group or change the group's traffic allocation. A governance gap in the exclusion group management is a common source of validity problems: two product teams each believe they have 50% of traffic for their experiments, but both experiments are in the same mutual exclusion group, so only 50% of traffic is actually available for either. The conflict is discovered when the first team launches and the second team's analytics shows an unexpected drop in their experiment's traffic — mid-experiment, after data has been collected at the wrong traffic allocation.

4. Metric hierarchy and guardrail policy

Document the metric hierarchy: the primary metric (the one thing the experiment is designed to move), the secondary metrics (supporting evidence, not decision-making criteria), and the guardrail metrics (metrics that must not regress beyond a threshold or the experiment is stopped regardless of primary metric results). The metric hierarchy is a product decision encoded in the infrastructure: the analytics tooling should enforce the hierarchy by surfacing the primary metric prominently, flagging guardrail metric regressions automatically, and preventing the experiment from being called a winner if any guardrail metric has regressed beyond its threshold. Without the hierarchy enforcement, PMs will rationalize guardrail regressions as acceptable when the primary metric is positive — "engagement went up 15%, the session duration regression is probably within noise" — producing a pattern where positive primary results are shipped with unacknowledged side effects.

Document the guardrail thresholds with explicit rationale for each threshold value. A guardrail on page load time (must not regress by more than 200 milliseconds at P99) is a business decision: a 200 millisecond P99 regression is the threshold at which previous experimentation or research shows a measurable impact on user engagement or conversion. The 200 millisecond value should be sourced from the product's own data (a regression analysis of user behavior against page load time percentiles) or from the literature (Amazon's 100ms-equals-1%-revenue-loss finding, adapted to the specific product's conversion elasticity). A guardrail with an unanchored threshold ("must not regress by more than 5%") is not interpretable as a business decision — 5% of what, and why 5%? The guardrail thresholds should reference the research or data that supports them. This connects to the API versioning decision record: if an experiment introduces a breaking change to the API behavior (changing an endpoint's response format as part of a treatment) the API's backward compatibility guarantee is the implicit guardrail — the experiment cannot ship if it breaks any consumer of the API. The decision record should document how API compatibility is checked as part of experiment winner criteria, especially for experiments that modify backend behavior.

Document the novelty effect policy explicitly as part of the metric hierarchy. Some product metrics are inherently susceptible to novelty effects (engagement metrics on new UI elements) while others are less susceptible (task completion rates, error rates, API latency). For metrics known to be novelty-susceptible, the experiment design should include a novelty washout requirement — a minimum exposure count or time period before the user's metric data is included in the analysis. A user who encounters the new onboarding flow for the first time will have novelty-driven behavior in their first 3 days; including their first-3-day data in the primary metric calculation conflates the novelty response with the sustained treatment effect. A user-level novelty washout (include only data from users who have been in the experiment for at least 7 days) separates the novelty spike from the steady-state effect. The washout requirement reduces the effective sample size and extends the minimum run duration — these tradeoffs must be understood before the experiment is launched, because discovering that the washout requirement makes the experiment 3x longer to run than expected is a planning problem best addressed in the design phase.

5. Observability instrumentation for experiment validity

Document the required logging and metrics for each active experiment. The minimum observability for an experiment includes: assignment logs (for each assignment event, the user identifier, the experiment identifier, the variant assigned, the timestamp, and the targeting attributes used), exposure logs (for each time a user was actually exposed to the experiment treatment in the product, distinct from the assignment event — a user may be assigned to treatment but never see the treatment if the feature is behind a conditional display), outcome logs (for each user, the value of the primary metric and all guardrail metrics during the experiment window, with timestamps), and configuration change logs (every change to the experiment configuration — traffic allocation changes, targeting rule changes, experiment start and stop events — with the author, the reason, and the timestamp). The combination of assignment, exposure, and outcome logs is required for valid analysis; experiments with missing assignment logs cannot be analyzed with user-level control, and experiments with missing exposure logs cannot distinguish between the set of users who received the assignment and the (potentially smaller) set who were actually exposed to the treatment.

The novelty detection metrics should be built into the observability layer: for each experiment, compute the primary metric value by day-in-experiment for each variant (day 1, day 2, ..., day N) and surface the trend. A strong day-1 spike that decays monotonically is a novelty effect signal; a stable primary metric value from day 3 or 4 onward indicates that the steady-state treatment effect has been reached. The novelty effect trend should be visible in the analytics dashboard before the minimum run duration is reached, so that the analyst can confirm the washout has completed before the experiment is called. An experiment where the primary metric is still declining at day 14 has not reached steady state and should not be stopped even if the minimum run duration in the policy is 14 days — the minimum run duration is a floor, not a ceiling, and the novelty effect trend provides evidence for extending the experiment beyond the minimum. This connects to the observability platform decision record: the experiment observability requirements (per-user, per-day metric aggregations across all concurrent experiments) generate a data volume that scales with the number of concurrent experiments times the daily active user count times the number of metrics tracked. A team running 15 concurrent experiments with 50 metrics each on a 100,000 DAU product generates 75 million metric observations per day in the experiment data pipeline. The observability platform must be sized for this volume, and the storage retention policy for experiment data must balance analytical completeness (longer retention enables retroactive analysis of past experiments) against storage cost (longer retention means more data to store and query).

Document the Sample Ratio Mismatch (SRM) check as a required quality gate. An SRM occurs when the observed ratio of users in the control and treatment groups differs significantly from the intended traffic allocation. A 50/50 experiment that produces 51,000 users in control and 49,200 users in treatment has an SRM: the chi-squared test for equal proportions will produce a p-value below 0.001, indicating that the assignment mechanism is not producing the expected balance. SRMs are caused by bugs in the assignment logic (a condition that prevents some users from being assigned to the treatment), by data collection differences (the logging pipeline drops events differently by variant), or by interaction effects with other experiments (a concurrent experiment is filtering users in a way that differentially affects one variant's population). An SRM does not invalidate the experiment outright, but it indicates that something unexpected is happening in the assignment or logging pipeline, and the source of the imbalance must be understood before the experiment results are interpreted. The SRM check should run automatically at experiment launch and continuously throughout the run, with an alert when the SRM chi-squared p-value crosses a threshold (typically 0.001). An experiment with an active SRM alert should not be stopped as a winner or loser until the SRM is resolved, because the unresolved imbalance means the user populations being compared are not random samples from the same underlying population. Document the SRM resolution procedure: who is responsible for investigating an SRM alert, what the diagnostic process is, and what the criteria are for declaring an SRM resolved versus an experiment invalidated. Without the procedure, an SRM alert produces a paralysis — the experiment cannot be stopped as valid and cannot be dismissed as invalid — that blocks the product team until someone in the organization has enough statistical context to make the call.

What to do with the A/B testing infrastructure decision record

The A/B testing infrastructure decision record's primary audience is the engineer who implements the experimentation platform, the data scientist or analyst who designs experiments and interprets results, and the PM who commissions experiments and acts on their results. The engineer needs the assignment model and the hash function specification to implement consistent assignment correctly the first time, rather than discovering the unit mismatch problem when the first important experiment fails to detect a real effect. The data scientist needs the statistical framework and stopping rule policy to design valid experiments and to push back on requests to stop experiments early or to change the primary metric after results are visible. The PM needs the minimum run duration, the sample size constraints, and the rollout granularity ceiling to set realistic timelines for experiment-driven product decisions — specifically, to understand that a product question that requires detecting a 5% improvement in a 15%-baseline metric requires several weeks of runtime and cannot be answered in 3 days regardless of how many engineers are allocated to implement the treatment.

If the A/B testing infrastructure decision record was not written when the experimentation platform was built, the most useful reconstruction starts with an audit of the assignment model. Review the code or configuration that implements variant assignment: what identifier is used, where the assignment call is made, and whether the assignment is deterministic and persistent. Run the SRM check on the 10 most recent experiments: if the control and treatment groups are systematically imbalanced, the assignment model has a bug that has been producing invalid results for all experiments since it was deployed. Run the peeking audit: for each experiment that was stopped early (before the originally-planned end date), compute the p-value that was observed at every day from experiment start to stop date. If the early-stopped experiments show a pattern of borderline-significant results at early looks that would not be significant under sequential testing boundaries, the peeking problem is confirmed and the historical results from early-stopped experiments should be treated as having a higher-than-nominal false positive rate. The ADR template provides the structure for documenting the current state as a decision — recording what was built, the implicit rationale behind the choices, and the consequences that have been observed.

The WhyChose extractor can surface A/B testing infrastructure decisions from ChatGPT or Claude conversations during the platform design phase — the discussion where the hash function was chosen, where the statistical threshold was debated, or where the team decided to use a third-party experimentation platform rather than building in-house may contain reasoning that was never formalized into architecture documentation. The decisions never written down essay covers why experimentation infrastructure decisions are particularly likely to be missing from decision records: they are made by data scientists or analysts, not engineers, and so they often live outside the engineering decision record conventions that the team uses for architectural decisions. The result is a split — the engineering ADR log covers the database, the API design, the deployment pipeline, but does not cover the statistical framework under which all product decisions are validated. A complete decision record archive captures both the infrastructure and the validity model that determines what the infrastructure can reliably conclude.