The card at a glance
The model card is the stats read the XP page serves, re-shaped — nothing on it is re-derived, so the card, the XP page, the Slack unfurl and the share link always agree. These are the words on its header, badge and footer.
Also shown as Not live, Too early to read, Getting first volume, Trending up, Trending down, Flat so far, Nearing auto-stop, Winner, Losing, No difference, Auto-stopped
Where the experiment sits on a five-step ladder: Not live → Too early → First volume → Reading a lean → Called. The badge, the coloured dots, the card border and the progress bar all show the same phase.
How it is computed
A called experiment
is its verdict (Winner / Losing / No difference / Auto-stopped). Only a
Pending one is placed on the ladder:
Too early when no challenger has traffic against the control, the lift is undefined, the smaller arm is under 10
expected conversions, or under 5% of the resolving sample;
First volume under 25% of it; otherwise a lean is reported —
Trending up above +3% on the primary metric,
Trending down below −3%,
Flat so far in between. Trending down becomes
Nearing auto-stop when the guardrail monitor says the brake is one gate away. The top rung of Pending is
Confirming a win / no difference / a loss: the read already calls something, and the badge counts how many consecutive hourly checks it has held toward the
24 that make it a call.
How to read it
Below "Reading a lean" any direction is noise by construction — the sentence beside the badge says so. A lean is a direction, not a call; only a called experiment should change what you do. "Confirming" means the numbers say one thing right now and the machine is waiting to see the same thing tomorrow.
Also shown as progress bar, ~N days at this pace
How far the smaller arm is towards the per-arm sample size the experiment needs before its result can be called.
How it is computed
smaller-arm visitors ÷
required visitors per arm, capped at 100%. The ETA divides the remaining visitors by the smaller arm's pace (visitors ÷ days elapsed, with a half-day floor so a one-hour-old experiment does not project a thousand-visitor day). An ETA past a year is not shown.
How to read it
Progress is measured on the smaller arm because a bandit starves a losing arm — the control can have plenty of traffic while the challenger is still short.
Total exposed visitors across every arm (see Visitors) and the day of the run, counted from the first exposure ("first day" on day one).
How visitors are split between arms. random is a fixed even A/B split, drawn in the visitor's browser (see client-side allocation). bandit shifts traffic towards the arm that is doing better (see Thompson sampling), drawn on the server. optimizely means Optimizely assigns the bucket and Bandit only measures.
How to read it
Under bandit an uneven split is the design, so "% of traffic" per arm is expected to drift and the sample-ratio check in Investigate is skipped. Under random an uneven split is a bug.
Also shown as drawn in the browser, arms, reported allocation
For a random experiment the arm is drawn in the visitor's browser — a stable hash of the experiment id and the visitor id over the experiment's published arm list — and applied at once; the browser then reports the draw to Bandit, which records it for a new visitor and answers the visitor's existing arm otherwise.
How it is computed
The arm list the browser draws over keeps DISABLED arms, so turning an arm off never moves anyone else's bucket: a new visitor who draws the disabled arm is simply outside the experiment (control, uncounted), rather than redistributed to the surviving arms. When Bandit's answer to the report names another arm — a visitor assigned before the browser drew, a declared winner — that page load is left uncounted and the browser renders Bandit's arm from the next load on. A declared winner is published to the browser, so everyone gets it without a draw. Pausing an experiment stops a browser's draw only once the brand is republished (the console does this on pause); a paused-but-published experiment is shown for one uncounted load per browser.
How to read it
Nothing in the readout changes: Visitors are still the ALLOCATE exposures, and a visitor is never counted in an arm Bandit answered differently. The draw removes the select round trip from the page's critical path, so a treatment lands before the veil lifts instead of after it.
The challenger the verdict is judged on: the one with the largest winsorized RPV lift against the control right now (the largest absolute lift when the control has no baseline). Violet chip on the card, violet-tinted row in the per-variant table.
How to read it
It can change between reads while the experiment is young. Every lift, CI and guardrail figure on the card and in the impact banner refers to this arm versus the control.
Model cardXP propertiesInvestigate
Also shown as Why KEEP?, confidence: high / medium / low
The verdict's recommendation. KILL: the CVR guardrail is significantly breached, or the experiment has converged with no upside left to wait for. CONTINUE: not yet converged, or the control has no baseline to compare against. KEEP: a converged Win or Super win.
How it is computed
Derived only from the
verdict label and the sign of the lift. Confidence is HIGH for a breach or a win, LOW when the control has no baseline, MEDIUM otherwise.
Shown only once the experiment has converged. SHIP on a significant Win against a live control; REVISE whenever the lift is not established — iterate on the concept rather than bank an unproven lift.
Model cardXP propertiesInvestigate
One sentence telling you what to do: kill, stop and revise, keep running, ship, or don't scale yet. It is a restatement of KEEP / KILL / CONTINUE and SHIP / REVISE, not an extra signal.
Copies a link to a snapshot of this card. Pasted into Slack it unfurls with the treatment strip. It is the same server-assembled card, so it cannot say anything the XP page does not.
Who is counted: cohort and window
Every rate on these pages has the same denominator — exposed visitors — and the same definition of when a visitor and a conversion belong to the experiment.
Model cardXP propertiesInvestigate
Also shown as impressions, exposures, recruits, exposed visitors
Distinct visitors who were allocated into the experiment (an ALLOCATE event with this experiment's key), less known bots (crawler networks and Chrome on Linux, from 2026-10-13) and resellers. Each visitor counts once, in the arm of their first allocation. Every rate on the page uses this denominator.
How it is computed
Visitor identity is userId, else anonId, else sessionId — in practice the browser's own id (csc_bandit_uid, local storage). First-touch: a visitor who later sees a different arm still counts in the first one — which is also the arm the allocator keeps serving them. Known gap: a visitor anonymous at exposure and logged in at purchase does not share a key, so that conversion is dropped.
How to read it
Not GA4 users, and not meant to match them. A visitor here is a browser's FIRST exposure in the window; GA4 counts everyone active on the days shown, by the Mountain-time day, and never sees the ~15% of shoppers who block it. With the bot rules, Visitors ran 0.9–1.2× GA4 users on the two Backcountry experiments measured (2026-10-08..11); before them, 1.5–2× across its page-load experiments. A date range on the page counts first exposures inside it; no range counts everything since activation (at most 90 days).
Also shown as share
The arm's visitors as a share of the experiment's visitors — realized share, not the configured weight.
Also shown as trafficShare, sampled in, % of matching visitors
The fraction of the visitors who match the experiment's pages that are sampled INTO it. The rest see the untouched page and are not in the cohort at all — no allocation, no exposure. Blank means everyone who matches.
How it is computed
Applied before the arm split, so 10% over two random arms is 5% control / 5% treatment. Membership is a stable hash of the experiment id and the visitor id: sticky per browser, salted per experiment (two 10% experiments sample different tenths), and monotone — raising the share keeps every sampled visitor and adds more; lowering it drops visitors who already have data. A QA force or preview link always renders regardless. Two experiments on one page that must not overlap name the same traffic layer (see below).
How to read it
Per-visitor rates are unaffected; exposures are that share of the page's traffic, so a 10% experiment takes ten times as long to reach a powered read. The brand's Coverage score measures what was actually exposed, so a sampled experiment counts for its share, not the whole page.
Also shown as trafficLayer, trafficOffset, slice, layer
A shared bucket space for experiments that run on the same page: every experiment naming the layer hashes the visitor identically, and each takes an explicit slice of it — the visitors whose bucket falls in [offset, offset + traffic share). Slices that do not overlap put every visitor in exactly one experiment.
How it is computed
Without a layer each experiment is salted on its own id, so a 10% and a 90% experiment overlap on ~9% of visitors and miss ~9%. With the layer, [0, 0.1) and [0.1, 1) partition everyone. The write path refuses a slice that overlaps another ACTIVE experiment's in the layer (drafts and paused experiments never conflict), and an offset without a layer. Offsets are explicit, never derived from the other members, so pausing one experiment does not move another's visitors. Neither field changes what a sampled visitor is served, so neither is part of the config hash.
How to read it
Each experiment measures its own control inside its own slice, and its exposures are its share of the page — a 10% slice takes ten times as long to reach a powered read. Growing one slice into a neighbour's is refused until the neighbour is cut or paused.
Also shown as exposure window, conversion window, N days
The exposure window bounds who is in the cohort: from activation (or the date range on screen) to the end date, at most 90 days back. The conversion window is per visitor and open-ended: it starts at that visitor's own first exposure and any conversion from then on counts, however late.
How it is computed
Window days = whole days between the first and last exposure, plus one. Truncating the conversion side at the end would silently drop late conversions and bias CVR and RPV down, so it is never truncated. Its start is the visitor's first touch, not launch: the event stream holds every visitor's purchases, exposed or not, and a purchase made before a visitor first saw the experiment cannot have been caused by it. Until 2026-10-12 the readout counted from launch instead — on a month-old experiment about 14% of conversions were purchases made before that visitor's exposure — and every stored readout was recomputed under the new rule.
How to read it
A date range on the XP page scopes the exposure cohort. The auto-stop panel is scoped to the dates on screen too, while the monitor always evaluates the full run — so the two can disagree, and the panel says so.
Also shown as exposeOn, Counted only once this appears
An optional selector that defers the ALLOCATE beacon until the element first appears. The denominator becomes "visitors who reached the surface" instead of every matching page view.
How to read it
Experiment-level, so the control is gated identically. Rates on a gated experiment are not comparable with an ungated one's; Investigate flags it.
Also shown as intent to treat, ITT, assigned visitors, among exposed visitors, gate share
The arms compared by every visitor the server assigned to them, whether or not the browser ever recorded an exposure — the comparison that stays fair when an exposure gate opens differently in one arm. Computed once a day for gated, randomly allocated experiments.
How it is computed
Intent to treat: every assigned visitor in each arm (resellers out), their purchases since launch, the same lift arithmetic as the readout. Unbiased, but diluted by visitors who never reached the gate, so it reads toward zero. Among exposed undoes that with the gate share of the treatment arms — (treatment − control) ÷ (control − (1 − share) × what never-exposed visitors do) — assuming the gate opens alike in every arm and a visitor who never reaches it is untouched. Control's own share is never used: it is the number a gate bug bends.
How to read it
When the traffic split is off, read this instead of the exposed lifts: a gate that missed one arm's visitors (an attribute-only control, 2026-10) can make a flat test read as a significant loss, and the visitors it missed cannot be recovered from the log. Daily, not hourly; experiments launched before the assignment table existed (mid-September 2026) do not get one.
Model cardXP propertiesInvestigate
Also shown as crawlers, bot traffic, Chrome on Linux, cro_allocation_excluded
An allocation from a known crawler network, or from desktop Chrome on Linux, is not counted: the visitor is in no arm, so out of Visitors, conversions, revenue and every cut. They are still served the experiment like anyone else — only the counting changes.
How it is computed
Judged per allocation on the request's address and User-Agent, never the page. Networks: Googlebot (66.249.64.0/19), Bingbot (four /24s), Meta's crawlers and link-preview proxies, Applebot (17.0.0.0/8), an AWS Singapore crawler (47.128.0.0/16) and Huawei Cloud (110.238.0.0/16) — each verified by reverse DNS or by traffic that was all single-page, never-buying and invisible to GA4. Chrome on Linux is headless Chrome's default and the shape of most automation; Android and ChromeOS are not Linux here, and Firefox or Chromium on Linux still count. The allocation is kept as a cro_allocation_excluded event with the reason, so the exclusion can be counted and undone. Applies from 2026-10-13: allocations recorded before then are still counted.
How to read it
Crawlers keep no storage, so each fetch was a new visitor: before the rule they were 35–50% of the Visitors on the two Backcountry page-load experiments measured, diluting CVR and RPV by up to 2× — equally in every arm, so lifts were unbiased. The Chrome on Linux rule also drops real Linux shoppers: 0.4% of buyers on one experiment, 4% on another (2026-10-08..11). An experiment running across 2026-10-13 counts bots before that day and not after, so its daily visitors step down; read lifts inside one side of the date when it matters.
Also shown as excluded resellers, Outliers
A visitor with any single order containing more than 2 units of the same base style at $15 or more per unit is dropped from the cohort entirely — numerator and denominator — before anything else is computed.
How it is computed
Grouping is by GA4 item_id, which on our brands is the base style (size and colour live in item_variant), so "20 of the same jacket in different colours" is caught. The $15 floor keeps sock and chain-lube stock-ups in. Applied whatever the success event is, because a reseller is not a shopper.
How to read it
Investigate warns when more than 2% of the cohort was excluded — the read is then about who was dropped as much as who stayed.
Also shown as freshness, a refresh looks overdue, Refresh now
When and how the numbers were computed. Hourly rollup: the whole-experiment aggregate the guardrail sweep recomputes every hour (the normal path). Daily rollup: day-grain rows summed to answer an explicit date range. Live query: the raw events were queried when the page loaded.
How it is computed
A rollup older than 6 hours is refused and the read falls back to live; the allocator trusts a rollup on identical terms. A rollup whose success-event fingerprint no longer matches the config is treated as absent. Once an experiment is no longer active its numbers are frozen and the age no longer matters. "Overdue" = a rollup older than 1.5 hours on an active experiment.
How to read it
Refresh now recomputes the hourly rollup instead of waiting for the sweep. Opening a page never advances the auto-stop confirmation counter — a stats read is not an observation.
Conversions and value
What counts as a conversion, and what a conversion is worth, are properties of the experiment. Every row, CI and significance is computed from the configured success event — not necessarily from purchases.
Model cardXP propertiesInvestigate
Also shown as Conversion, Purchases, Optimizing for, Conversions (i)
The event that makes an exposed visitor a converter, counted once per visitor and only at or after that visitor's first exposure. The default is a purchase (any order), valued at its order revenue. An experiment can instead name any event type with property predicates (e.g. an add-to-cart, a page reached), valued by count, by order revenue, by an event property, or by a fixed amount.
How it is computed
There is no unvalued state: a count-unit metric is worth exactly 1 per converting visitor, so per-visitor value for a count metric is the conversion rate under another name. ALLOCATE and CRO PAGE EXPOSURE can never be the success event (they are traffic, and would report a rate near 100% that measures nothing).
How to read it
The column header on the card takes the event's label ("Purchases", "Add to carts"). When the metric is a count, revenue columns disappear and "Per visitor" replaces RPV.
Up to four extra events that add to a visitor's value but never make them a conversion.
How to read it
This is why value per visitor can be above zero while conversions are 0: a visitor can buy anything (value) without doing the configured success action (conversion). The Conversions (i) hint on the XP page spells out both events.
Model cardXP propertiesInvestigate
Also shown as Conv., N attr (p%)
Converting visitors, not events. A visitor with two orders is one conversion. The sub-value "N attr" is the attributed subset and its share.
Model cardXP propertiesInvestigate
Also shown as Purchases rate, N% under the conversions count
Conversions ÷ visitors for the arm. 0 when the arm has no visitors.
How to read it
CVR is the
guardrail metric: a significant drop in it is what KILLs an experiment, whatever the revenue says.
Also shown as Value (event), Realized so far
The sum of each cohort visitor's success value: order revenue for the default, or the configured property, fixed amount, or 0 for a count source. The totals grid hides it entirely for a count metric.
Also shown as RPV, Per visitor, revenue per visitor, winsorized RPV
Revenue per exposed visitor — every visitor in the arm, zeros included — with each visitor's value capped at the 99th percentile of buyers so one huge order cannot swing the figure. This is the primary effect-size metric the verdict is judged on.
How it is computed
The cap is one number per experiment, computed over converters across all arms (the footnote under the per-variant table prints it). The totals-grid RPV is the raw, uncapped revenue ÷ visitors, so it will not equal the per-arm winsorized figures.
How to read it
For a count metric "Per visitor" is the share of visitors who converted — the CVR column by construction — and the cap has nothing to trim.
Also shown as attributed, Bought shown item, items tracked, Attributed (i)
The subset of conversions that contain a style this experiment showed the visitor. Shown = whatever the experiment's ops report (a tracked section or a widget); matched on the purchase event's item_id, the base style, so any size or colour of a shown style counts. Attributed value is price × quantity of the matching lines only, not the whole order.
How it is computed
A visitor's seen set is every SKU any of this experiment's widget events recorded for them, anywhere in the window — they need not have used the widget itself. A matching line counts only when bought at or after the visitor's first exposure, like every conversion. Attributed rates keep the parent's visitor denominator. Resellers are removed and attributed value is capped at the p99 of attributed buyers. Styles a widget would have shown (counterfactual rows) are deliberately not in the seen set.
How to read it
"Attributed metrics not tracked" with a reason means the block is null, never zeros: zeros would claim "we showed people things and they never bought", which is a claim, not a gap. The All conversions / Attributed only toggle decides which set the CI and Sig columns use; the verdict is always judged on all conversions. Attributed figures are not cut by device.
Lift, intervals and significance
Two lifts run side by side on every page: a CVR lift (the guardrail) and an RPV / per-visitor lift (the effect size). Both are the challenger relative to the control, with a 95% interval.
Model cardXP propertiesInvestigate
Also shown as CVR lift, RPV lift, Per-visitor lift, ATC lift, Purchase lift, lift vs control
(challenger − control) ÷ control, as a percentage. The control row reads baseline.
How it is computed
Undefined — shown as — or ∞ — when the control is zero: ∞ when the challenger moved off a zero base, — when neither arm registered anything. It is deliberately not rendered as +0.0%, which would read as "no difference", the one thing it is not. A lift is only computed for a challenger with traffic against a control with traffic; otherwise a zero-traffic arm would report a misleading −100%. A disabled arm's numbers stop at the moment it was disabled, so its lift compares it with the control as it stood at that moment — control's visitors first exposed by then and their conversions up to then — never with control's still-growing totals, which kept converting for days the arm could not (one arm read −22% that way and −3% at its disable). A date-range view, or a non-purchase count metric, cannot cut control there and compares with control's full range instead.
How to read it
Colour is by sign: green up, red down, muted when unmeasured. On the model card a lift that is measured but not significant is left uncoloured. Significance is a separate chip or column, never a colour.
Also shown as delta vs control, per-visitor delta
challenger − control in the metric's own units (percentage points of CVR, dollars of RPV). Always defined, because a difference needs no baseline — it is what the $ per 1,000 visitors and realized impact figures are built on.
Model cardXP propertiesInvestigate
Also shown as interval, lo to hi, RPV lift 95% CI, no interval, no baseline
The range the relative lift plausibly sits in: the point estimate ± 1.96 standard errors. Null whenever the relative lift is.
How it is computed
Delta-method normal approximation. CVR: SE from each arm's binomial variance. RPV: SE from each arm's measured (winsorized) per-visitor variance — not the old Poisson shortcut, which treated revenue as a count of $1 events and understated the error by an order of magnitude.
How to read it
An interval that
clears zero (both bounds the same sign) is what "significant" means on the card and in every rule below. The interval's width, not the point estimate, is what says whether a lean is real; under the
np floor the interval is not describing anything.
The horizontal bar in the CI column: the 95% interval drawn against a zero line shared by every row.
How to read it
Grey when the interval crosses zero (not significant), green when it sits above, red when below. When there is no interval nothing is drawn — a zero-width bar on the zero line would read as a tight, confidently null effect.
Also shown as yes / no, sig chip, Not yet significant, The interval clears zero
Whether the CVR difference between the arm and the control is statistically significant.
How it is computed
Two-proportion z-test on conversion rates with a pooled rate; two-tailed p; significant when p is below 0.05. The card's "sig" chip instead asks whether the RPV lift interval clears zero — the two usually agree and can differ at the margin.
How to read it
Significant is not the same as converged. "The interval clears zero, but the sample is still short of a powered call" means exactly that: direction is trustworthy, magnitude is not.
Also shown as about N expected conversions, Smaller arm has 30+ expected conversions
Smaller-arm visitors × the control's conversion rate: how many conversions the smaller arm should have by now. Nothing that acts on or announces a significant interval does so below 30.
How it is computed
Below the floor the delta-method interval has a degenerate tail: a zero-conversion challenger's SE collapses to zero and its "95% CI" is exactly [−100%, −100%], which reads as established catastrophic harm on a few dozen visitors (this happened in production on 2026-09-01). The floor gates the declaration — Loss label, KILL, breach episode, Slack — so they stay quiet together. The phase ladder uses a softer floor of 10 to report a lean, because a direction is a softer claim than a breach.
Verdict and convergence
The verdict is deterministic — the same numbers always give the same read, and the read becomes a call only once the guardrail monitor has seen it hold for 24 consecutive hourly checks. Effect size is the winsorized RPV lift of the primary challenger; the guardrail is that challenger's CVR lift; convergence is the front-runner's bet resolving, confirmed over a day.
Model cardXP propertiesInvestigate
Also shown as Terrible loss, Loss, Pending, Neutral, Win, Super win
The six-way call, judged in this order — the first that matches wins.
How it is computed
Each read calls one thing, in this order:
loss when the CVR guardrail is
breached;
win when the front-runner's bet is
worth backing;
neutral when it is not worth backing; nothing while the bet is undecided or the smaller arm is under 30
expected conversions. The read becomes the label only after
24 consecutive hourly checks agree:
Loss (
Terrible loss if the CVR drop is worse than −10%),
Win (
Super win if the RPV lift's lower bound is above +10%), or
Neutral. Until then it is
Pending, and the keep hint says what the read calls and how long it has held.
How to read it
Pending is the honest state for most of a run, and for the first day of every run: a read that resolved eight hours after launch on a −11% to +40% interval is a read, not a call. The auto-stop does not wait for the label — it reads the raw breach and acts on its own three confirmations.
Region of practical equivalence: a relative RPV lift within ±3% is treated as "practically zero". It is also the ±3% that separates Trending up / Flat / Trending down on the phase ladder.
Model cardXP propertiesInvestigate
Also shown as 18% on a null change, a change that does nothing reads this well N% of the time, about 1 in 5
How often a change that does nothing would read at least this far ahead at a single look — the one-sided tail of the no-effect curve at the observed lift. "18% on a null change" means about one null test in five looks exactly like this.
How it is computed
Computed from the same interval as
chance ahead: 1 − Φ(|lift| ÷ SE). It is the number the bet framing had been hiding. "Worth backing" (10:1 reward:risk) sits 0.90 standard errors above zero, where a null change reads this well
18% of the time; 90% ahead is 10%; the interval clearing zero is 2.5%. It is printed under every chance ahead, in every worth-backing sentence, and beside a called Win — so a cheap positive-expectation bet and an established result stop looking alike. It is a
one-look figure: the monitor reads hourly for weeks, so a flat arm meets a given bar at some point more often than the per-look number says, and the
24-check confirmation filters jitter, not a stable lucky read.
How to read it
Read it beside the label, never instead of it. A Win carrying "18% on a null change" is a positive-expectation bet whose evidence a null would produce one time in five; a Win carrying "<1%" is established. The cheaper the change is to ship, the more a high number is tolerable; the more the result will be cited, the less. The case it was added for: a one-word PDP header rename on Steep & Cheap read 10:1 at 504k visitors per arm with a flat add-to-cart step (2026-10-07) — a reader seeing "18% on a null change" beside that 10:1 would not have called it a result.
Also shown as Yes / Not yet, Visitors on smaller arm N / M
Whether the result can be called. Three gates, all required: the smaller arm has 30 expected conversions (so its interval describes something), the front-runner's bet has resolved — worth backing or not — at the lift it is showing now, and that same read has held for 24 consecutive hourly checks. There is no calendar floor; the 24 checks are the only clock, and they are a stability requirement, not a duration.
How it is computed
An undefined lift (control with no baseline) can never converge, however much traffic the arms have. A lift sitting exactly on a bar cannot resolve either. The "Time to a call" panel shows the sample the bet resolves at and, once resolved, the checks held so far.
Also shown as held N of 24 hourly checks, Confirming a win, not a call yet
The guardrail monitor checks every active experiment hourly and keeps a run of consecutive checks on which the verdict called the same thing (win / neutral / loss) about the same challenger. The label is a call only when the run reaches 24. It is the auto-stop's own confirmation mechanism with the bar set at a day instead of three hours.
How it is computed
Three things start the run over: a check that calls something else (or nothing), the front-runner changing, and a human editing the experiment after the run began. A missed hour is not a reset — it counts checks, not hours. A read between checks reports the count as of the last check.
How to read it
The 24 checks run on cumulative data an hour apart, so they are not 24 independent reads: the lead has to survive the sample roughly tripling without once dipping under its bar. It filters a day-one lucky read that fades; it does not cover day-of-week, and a coin flip that stays lucky for a day still gets called — the interval and the sample are beside the label for exactly that reason.
Also shown as results veil, enough to make a call, not enough to make a call yet
Whether the readout has enough to make a call yet. Veiled: not yet — the badge says what the call is waiting for and roughly how long at the current pace. Unveiled: since the date shown, the dashboard has had enough. Ended veiled: the experiment stopped before it got there, so its numbers are directional only. A status beside the numbers — nothing is hidden on it.
How it is computed
It lifts on the same rule the wrap-up notice fires on, never a rule of its own: a
confirmed call (the same read held for 24 hourly checks), a stalled interval (the smaller arm past its powered sample and the interval narrowing by under 10% over 7 days), or a declared winner. Once a wrap-up has gone out it stays lifted. While veiled, it waits on the first unmet gate in order: the call streak if a read already calls something, else 30
expected conversions on the smaller arm, else the visitors the current lift needs to resolve — then a day of checks. The ETA is at this lift and this pace; an early lead is the lucky one more often than not.
How to read it
Read a veiled point estimate as a guess inside its interval, red or green. "Ended veiled" is the stamp a re-run comparison needs: an experiment paused before its call did not disprove the next one.
Also shown as metric concordance, where the lift lives, value per buyer
Whether the funnel steps a change has to move on its way to the headline moved the same way the headline did, per challenger. Agree: add to cart or the purchase rate leans the headline's way and neither leans against it. Disagree: one of them moved the other way with its 95% interval clear of zero. Unexplained: the headline leans, but neither leans with it (or one leans against without clearing zero). No direction: the headline itself has none yet.
How it is computed
The headline has a direction once its
chance ahead is at least 80% (up) or at most 20% (down). Each voting step is read on the chance it moved the same way, from its own interval: 80%+ leans with it, 20% or less leans against. Only add to cart (any product) and the purchase rate vote. For a purchase experiment RPV = add-to-cart rate × cart→purchase × value per buyer exactly, so the relative lifts multiply and each factor's
share of ln(1 + RPV lift) says where the lift lives; value per buyer is what is left after the two rates and has no interval of its own. Cart→purchase is shown, never a vote: more carts from less-committed shoppers pull it down while carts and buyers both rise. For any other success event the headline is that event per visitor and the steps are the add-to-cart and purchase that follow.
How to read it
Direction only, by design. Whether a lift should live in value per buyer depends on the change — an add-on attach legitimately raises it, a header rename should not — so the shares are shown and the hypothesis judges them. The case it was built for: a header rename on Steep & Cheap read 10:1 with add-to-cart dead flat and the whole lift in cart→purchase, where a header cannot act.
Also shown as how this was computed, GA4 odds, derivation
Every step from an arm's observed numbers to the reward:risk odds, with the numbers plugged in: visitors n, the winsorized value per visitor x̄ and its variance s² for each arm; the two standard errors; the lift L and its standard error (delta method, both arms' noise); the 95% interval; t = L ÷ SE; chance ahead Φ(t); expected gain G = L·Φ(t) + SE·φ(t) and loss Λ = G − L; reward:risk = G ÷ Λ; chance of harm, of a real win, and on a null change.
How it is computed
Built by the same functions the readout uses, from the moments each comparison actually used — a disabled arm's against control as it stood when the arm was turned off. Odds from GA4 (GA4 report → Compute) rebuilds those moments from the brand's GA4 export: GA4 users (first touch on the first cro_allocation for the experiment), purchases at or after first exposure one per transaction, resellers removed, revenue capped at the 99th percentile of GA4 buyers — then runs the same derivation. It reads ~9 GB of export per day of window, so it runs only on a click, shows its estimate first and is cached for half an hour.
How to read it
A reader can stop at any hop they already trust, or check one by hand. Hold the GA4 odds against Bandit's for direction and rough size, not head counts: GA4 filters known bots and only holds an exposure once the storefront tag forwards it, so its users run below Bandit's visitors on a page-load experiment.
Model cardXP propertiesInvestigate
Also shown as sample-size target, Sample N / M, Needs ≥N visitors/arm
The visitors each arm needs to detect the target MDE on the control's observed conversion rate, at 80% power and a two-sided 5% alpha.
How it is computed
Standard two-proportion sample-size formula on baseline p and p × (1 + MDE). Falls back to 15,000 per arm when the baseline is 0 or 1. Because it is anchored on the observed baseline, the target moves a little as the control's CVR settles.
Also shown as assumed MDE, Expected magnitude, detect a N% relative lift
The minimum detectable effect the experiment is powered for: a relative lift in CVR. Pre-registered on the experiment as a percentage; 5% when none is set.
How to read it
A smaller MDE means a much larger required sample (roughly the inverse square). Pre-registering a realistic magnitude is the one lever an operator has on how long a read takes.
Also shown as power N%, currentPower
The design power for the assumed MDE at the current per-arm sample: the probability the experiment would detect a lift of exactly the target MDE if one existed. It reaches 80% exactly when the sample-size target is met.
How to read it
This is deliberately not post-hoc power for the observed effect, which is just the p-value restated and reads as a contradiction next to a "needs N more visitors" gate. (A legacy powerAnalysis block on the raw API still carries the post-hoc version; the console does not show it.)
Also shown as per 1k, normalized for extrapolation
The winsorized per-visitor value delta of the primary challenger versus control, × 1,000, with its 95% interval. For a count metric it is extra conversions per 1,000 visitors — same arithmetic, different unit.
Also shown as Projected monthly (event) lift, visitors/mo at the current exposure rate
What another month at the current exposure rate would bank: $ per 1k × (visitors per day × 30.44 ÷ 1,000), with the interval scaled the same way.
How it is computed
Visitors per day uses days elapsed with the same half-day floor as the phase ETA, so the two never disagree. Withheld below the "Reading a lean" phase, for count metrics, and for any XP that is not active (a paused test banks nothing next month) — a "$40k a month" on day one is a guess wearing a dollar sign. The CRO Overview headline ("Proj. monthly lift") and each line's $/mo are the same figure for the line's current XP, summed — so a line whose newest XP is paused or too early contributes nothing, and the headline is the sum of the running, readable XPs only.
How to read it
A forecast, not money in hand. The realized figure below is what has actually happened.
Also shown as net against control over N exposed visitors, realizedRange, Already banked, not a forecast
What the experiment has already cost or earned: for each challenger, its winsorized per-visitor delta vs control × its visitors, summed. The visitor count is every challenger's visitors (control excluded).
How it is computed
The total carries its own 95% interval, and each figure is only ever shown beside the interval for that same figure — the primary challenger's line has its own. The total's interval is not the sum of the per-arm ones: every challenger is compared against the same control, so the control's noise adds up across arms while each arm's own noise adds in quadrature. With one challenger the two intervals are identical. Winsorized on purpose — one bike order can flip the sign of a whole experiment. Null (shown as —) when no control resolves: 0 would read as "cost nothing", a measurement rather than the absence of one.
How to read it
Wherever the monitor or Slack quotes realized harm it must carry its interval ("−5,836 (95% CI −14,185 to +2,514)") — the point estimate alone is never a spelling.
Also shown as Suggested focus: dimension (push higher)
The verdict's draft of what to test next. A KILL proposes a fork (change the idea); KEEP or CONTINUE proposes a continue that pushes the primary variant's highest-scoring dimension further.
Guardrail, auto-stop and the monitor
The guardrail monitor sweeps every active experiment hourly, recomputes its rollup, and keeps three run-length ledgers: consecutive breaches, consecutive wins, consecutive trending reads. Everything in this section is read off that sweep.
Model cardXP propertiesInvestigate
Also shown as CVR lift vs control · 95% CI, Monitoring, No baseline, No breach, Underperforming, Nearing auto-stop, Auto-stopped
The primary challenger's CVR lift against the control, with its interval. Whatever the revenue says, an experiment that significantly lowers conversion is a Loss. The badge is one of six states, worst wins.
How it is computed
Monitoring: no challenger has traffic yet. No baseline: the control converted nobody, so the ratio has no denominator — undefined, not 0%. No breach: measured and fine — the interval does not put CVR significantly below control. Underperforming: measurably down, still serving. Nearing auto-stop: one gate short of a stop. Auto-stopped: the brake fired and the experiment is paused — only an actual fire earns this; a shadow-mode "would fire" cannot borrow the badge.
Also shown as guardrail breached, breached N consecutive checks
The whole 95% CVR-lift interval is below zero and the smaller arm has at least 30 expected conversions.
How to read it
This is the only auto-stop gate that asks about confidence. It is what turns the verdict into Loss / KILL; the remaining gates decide whether the machine acts on it.
Also shown as N of M met, Lift is computable, Drop is material, Powered for the observed harm, Breached on 3 consecutive checks
The ladder the monitor climbs before pausing an experiment. A check on a row means that condition for stopping is satisfied — evidence to pause, not an all-clear. Amber marks the one row still holding the stop back.
How it is computed
In order:
Lift is computable (lift and both bounds exist, and the primary variant is still configured).
Drop is significant (
breach).
Drop is material (
materiality).
Powered for the observed harm — reported, does not gate.
Smaller arm has 30+ expected conversions.
Breached on 3 consecutive checks (
confirmed). All gating rows met ⇒ fire. The brake has a second, faster route for an arm that has stopped working rather than one that is losing:
breakage.
How to read it
The panel is scoped to the dates on screen, a what-if; the monitor evaluates the full run. When the estimate is a gain, no gate is near firing and the unmet rows are what a stop would need, not what is holding one back.
Also shown as Breakage suspected, Dead arm, BREAKAGE: … stopped converting to cart
An arm whose visitors have stopped adding to cart — the plumbing failed, not the idea. Judged on the last 24 hours only: of the visitors first exposed in the window, the arm carted at most 20% of what control's add-to-cart rate predicts, on enough traffic for that to be a fact (20+ expected add-to-carts on the arm, 20+ carting visitors on control). Confirmed on 2 consecutive hourly checks ⇒ the monitor pauses the experiment, through the same switch, bypass and republish as the harm ladder.
How it is computed
It exists because the harm ladder is built to be slow — confidence interval, materiality, the np floor, three confirmations — which is right for a treatment that converts a little less and wrong for one that has lost its add-to-cart button (2026-10-01: a storefront redesign wrapped the buy box, three arms of a live PDP experiment hid it, and the ladder was still gathering evidence when a human noticed). A dead arm is obvious on a few hundred visitors. The control is never judged (a storefront bug in control is not the experiment's doing), nor a disabled arm, nor an arm with too little traffic. First check posts Breakage suspected to the brand channel with the numbers; second check stops. The companion in the browser is the protected-node fence: a hide that would swallow the add-to-cart button is refused and beacons cro_op_refused, so this rule is the backstop, not the first line.
How to read it
Read the detail's numbers against the arm's selectors on the live page (live-render, the cro_op_refused events): a breakage is a selector that no longer means what it did, and the fix is the selector, not the hypothesis.
Also shown as has run long enough, appears to be stable, having run this long, Wrapped up, Fan-out — where this idea has not run, Transfer matrix
The monitor's one notice that an experiment is finished, posted once per experiment per call to the brand channel. Two triggers: the read has been called and held for 24 consecutive hourly checks (confirmed), or no call and the primary challenger's interval has narrowed by less than 10% over the last 7 days while the smaller arm is already past the powered sample — the read is as good as this traffic will make it. The notice names the action the read supports (declare and ship / pause / pause), one or two variants worth trying next — the same idea built differently, proposed by Claude from the experiment's full definition and its read — and the fan-out: every other serving storefront, in any operating company, that has never run the concept, with what the idea is worth there. Every other storefront's channel also hears it once: "<brand>'s XP won / lost / came out neutral — think about how it could apply here", with that storefront's estimate.
How it is computed
The worth of a never-run cell is three measured factors multiplied: the pooled lift of the storefronts that ran it (inverse-variance over their intervals, shrunk toward zero by a 10% prior, so one noisy read cannot dominate), times the target storefront's visitors a month on that surface, times its value per visitor there — both read from the target's own experiments on the same page type (their exposure-gated denominator and their control's winsorized RPV), else from page views with no dollar basis, in which case the lift is stated and the money is not. Beside every estimate: risk — the source storefronts disagree, a neighbouring idea read the other way on the target, or the treatment is custom JS that cannot be copied as ops. The matrix is ONE across every operating company and is recomputed every three hours; a storefront whose serving is switched off counts as evidence but is never a fan-out target. The estimate a transferred experiment is created with is kept on it and scored against its realized read later (the matrix's calibration), which is the one honest number the platform will have on whether its reads generalize.
How to read it
A wrap-up is a suggestion, never an actuation: nothing pauses, declares or publishes on it, and it is worded as one — the read appears to be a win, consider pausing it — because a read is evidence, not a ruling. The number it quotes is the metric the read was made on: RPV (or value per visitor) for a win or a neutral read, conversion for a loss. A neutral or losing read is about this build at this traffic, not the idea — a wash here is not a wash everywhere, so a neutral read invites the other storefronts rather than warning them off, and only a loss carries the caveat to carry it where the surface differs. The suggested variants are a model's proposals, not results. A storefront the reads say it would cost (its whole estimated interval below zero) is named in the notice in those words but never invited in its own channel. When the notice names no other storefront it says why: every one has run it, the experiment has no concept (link its line), or the matrix has not been computed yet — never "nobody is missing it" on a matrix it could not read. A changed call (win → neutral under more traffic) earns a corrected notice. A bypassed auto-stop, a declared winner or a paused experiment gets none. Draft a fan-out on the other storefront from the XP page (Copy to brand) — the matrix proposes, an operator launches.
Also shown as −10% or more, or $4,000 lost
The harm is big enough to be worth stopping a test over: the CVR lift point estimate is −10% or worse, or (for a valued metric) realized impact is −$4,000 or worse with its whole interval below zero.
How it is computed
Tested on the point estimate, the same end of the interval the verdict reads. An earlier version tested the interval's optimistic bound and held a −28% breach with $5.5k banked for 19 hours. No progress bar on purpose — "68% of the way to material" invites reading a threshold as a countdown.
Whether the smaller arm has the sample needed to call a drop as large as the one observed (at least the 10% bar). Context only — it does not vote.
How to read it
The requirement is sized against the observed magnitude, so an estimate regressing towards the truth raises its own bar: a moderate, real, stable harm could never satisfy it, which is the exact case a brake exists for. The np floor took its place as the gate.
Also shown as N of 3 consecutive breached observations, Confirmed breaches counted by the monitor as of…
How many hourly monitor reads in a row have seen a breach. The stop needs 3. Any clean read, or any human touching the experiment, resets it to 0.
How to read it
Counted in observations, not hours, and only by the monitor: opening the XP page never advances it. This is the guard against a sequential-testing false alarm.
Also shown as governing mode, a met gate pauses the experiment, records only, never pauses
What a fully met ladder does. Enforcing: the experiment is paused. Shadow: the decision is recorded and alerted, never acted on. Off: the ladder is not evaluated.
How to read it
The page shows the governing mode — the mode of whichever tier actually acts on this row. Staging reads production's database in shadow while production enforces, so a page served by staging would otherwise say "shadow" about an experiment that production will stop.
Also shown as One condition short of an automatic stop
A breached, evaluable experiment that is either already material, or halfway to the money bar with a significant realized-harm interval. A notification, never a gate: being wrong costs a message, not an experiment.
Also shown as measurably ahead, not a final call, win, flagged N ago
The stop rule mirrored in the positive direction, on CVR: the lift's lower bound is above zero, the lift is +10% or better (or realized gain is +$4,000 or better with its interval above zero), the smaller arm has 30+ expected conversions, the guardrail is not breached — on 3 consecutive checks.
How to read it
Spoken as an opinion, never "winner": three consecutive reads an hour apart on cumulative data are close to one sample read three times. It gates nothing — no promotion, pause or reallocation follows. "The lead has since faded back inside the interval" is the flag being honest about that.
Also shown as trending harmful, not yet significant
On the card: the sign of the primary metric's lift beyond ±3%, at a phase where a lean is worth reporting. In the monitor: the magnitude question without the significance question — a CVR point estimate at −10% or worse (or −$4,000 realized) with 30+ expected conversions, while the interval still crosses zero.
How to read it
"Not yet significant" is load-bearing in that Slack line; without it, it would be indistinguishable from a breach. Trending and breached are mutually exclusive by construction — the more certain word wins.
The experiment was paused by the monitor but the CDN config still lists it. An Optimizely-allocated experiment keeps serving until the brand is republished.
Also shown as pending first scheduled run, just now / N min ago
When the hourly sweep last evaluated this experiment. The dot is red after a fire, amber on a shadow-mode would-fire, green otherwise.
Also shown as Declare winner, Declare winner and create PR
An operator declared this arm the winner. From that instant every shopper is served it — including the ones who had been in another arm — and measurement is closed for every arm: nothing after the declaration counts as an exposure or a conversion, so the numbers stop moving and stay the comparison they were. The experiment remains active because it is what delivers the treatment until the source change ships. Permanent: there is no un-declare; the ways out are pausing (everyone sees control) or duplicating the experiment. The control cannot be declared, and the declared arm cannot be disabled.
Also shown as not live yet, live for some shoppers, safe to retire
Shown once an arm has a storefront pull request. The shipped code names the experiment on every page render where it is live — only where its feature switch is on — and the engine stands down there and reports a sighting. Each hour the monitor compares the last 24 hours of sightings with the shoppers the experiment newly allocated. Not live yet: no sightings (not deployed, switch off, or rolled back). Live for some shoppers: the experiment is still the only thing giving everyone else the treatment — keep it running. Safe to retire: at least 200 shoppers a day report the change live, new allocations are at most 0.5% of that traffic, and that has held for three hourly checks in a row; one miss resets the count. It is never automatic: retiring is your click, because a switch rolled back afterwards would leave those shoppers with neither version.
Also shown as late apply, late-apply share, applied after the veil, PAGE COST: potential flicker on … of loads
A page load on which the experiment released the anti-flicker veil after the host's own timer had already shown the page — the arm landed on a page the shopper was reading, a change they could see happen. Measured per load from the client's sampled timing fact (revealAtMs against the host's veil cap, which the loader computes from data-start-time + data-hydration-timeout), and rolled up hourly per brand and device as such loads over loads that said either way. Control loads count too: their release is invisible, so the share is a ceiling on what shoppers could notice — and on a host whose timer fires before its React hydration lands (Backcountry: 500 ms timer, hydration at about 1 s) every arm that must wait for hydration is past the timer by construction. Read it with after hydration (the release minus the host's hydration signal: the engine's own cost once the DOM was safe to change) and the avoidable share (releases more than 100 ms after the page was both showing and hydrated): an arm landing within a beat of hydration is content arriving, not the engine being slow. In October 2026 the loads released after the timer converted about 30% worse than the ones released before it.
How it is computed
The hourly page-cost check warns at 25% and alarms at 40% of a device's sampled loads, or at 1.5× the brand's own 7-day same-hour median once a week of rows exists (never below 10%). A device-hour needs 300 sampled loads to be judged; a finding is suspected on its first hour and confirmed on the second consecutive one, and the brand channel hears it once, then once more when it recovers. The CloudWatch alarm watches the worst brand's all-device share over two hours at 40%.
How to read it
A climb here is delivery, not the arms: the bundle or config arriving after the veil deadline, a storefront release that pushed the bootstrap later, or the network. Read it with the hold p90 and the select-failure share for the same hour, then the latest storefront release and the bundle deploy time. Nothing is paused on this signal.
Also shown as potential flicker end, flicker end p90, after hydration, avoidable flicker, veil hold p90, hold, apply, select failures, cro_timing_hourly
What the shopper waited on the engine's account, per hour, brand, device and arm, from the client's sampled timing fact: shown at (when the page was displayed, on the navigation clock: the engine's release or the host's own veil timer, whichever came first — the one number for when the shopper saw the page; the clocks below are the engine's side of it), the potential flicker end (from the experiment's task start to the moment its change had settled: while the host's own timer still hides the page this is a veil hold, and past the timer it is how long the shopper could watch the page change — the host's timer, 500 ms on Backcountry, bounds the blank time whatever this reads), the apply (its own pre-paint ops), the network select, where on the page clock the release landed, LCP and long-task time, and the loader's boot clocks. Control included, so an arm is always read against its own control — and a control settles at its task start (an arm with no ops holds nothing; since 2026-10-20: before that it waited for the host's hydration signal like a gated op, so every control visitor's veil was held to the timer and a control read slower than its inert treatment), so a hydration-gated arm's whole wait reads as its own cost.
How it is computed
Bars on the global rows: potential flicker end p90 at 500 ms on desktop and 1,500 ms on mobile, judged per known device (a fact with no device stamp is crawler traffic and keeps its own unknown row, never judged and never pooled into all devices); the select-failure check is withdrawn until it has a per-page-load denominator (allocations are counted once per visitor, so the old share overstated failures about tenfold). Bars on an arm against its control, all devices pooled, with 200 sampled loads on each: a slow arm settles its change 100 ms later than control at the median or 300 ms later at p90 (the inert A/A's noise was 9–17 ms); a heavy arm spends 200 ms or more at p90 in its own op work — the engine's time inside the arm's ops once each op's target existed and a widget's data had arrived (apply_work_p90); the wait for a target to stream in or render at hydration, and a widget's wait on its data (its fetches, and the queue behind the page's own scripts between them), are the page's clock, not the arm's weight (until 2026-10-20 the bar read the whole apply, and a one-attribute inert arm tripped it on loads whose buy box had not parsed yet; until the same day's second change a widget's fetches counted as work, so the BC “goes great with” arms read 4.4–4.9 s at p90 of mostly waiting); page drift is 250 ms more LCP at p75 or 150 ms more long-task time at the median than control. Slow arm is a warning; heavy arm and drift are advisories — a badge, never a channel line.
How to read it
Two consecutive hours over a bar confirm it. A slow arm with a flat heavy-arm read is usually the platform (delivery) landing differently on the two arms by chance of timing, or a widget waiting on a fetch; a slow arm with a heavy apply is the arm itself. None of this stops an experiment; it says where to look before the conversion data can.
Also shown as engine p50, path p50, delivery, gate, path share, CroPathShare, scheduler lag, loop lag
Engine p50 on the card (path_p50 in the rows): the time from the host's head-parse to an experiment's release — everything the engine owned on that load — split into the edges the engine actually walks: delivery (snippet to the bundle's last byte), config (to the config's last byte; zero when the host inlined it), the gate (engine ready to the experiment's task start — the host's DOMContentLoaded, hydration and scheduler wait), select and apply. Each is read per sampled load from the engine's own marks on the navigation clock and rolled up hourly per brand, device and arm as a median and as the mean share of the path it owns.
How it is computed
The shares answer the operations question directly: which node adds the weight. A delivery share climbing is the CDN or the storefront head; a gate share climbing is the host page getting heavier or the scheduler busier (read it with the scheduler lag — the engine's event-loop lag: a 100 ms heartbeat's median lateness that the engine measures for its first ten seconds on the page, the delay a busy main thread adds to anything the engine schedules); a select share is the edge API; an apply share is the arm itself. The bundle and config fetch phases (first byte) appear only once the origin sends Timing-Allow-Origin; until then those two edges are whole fetches.
How to read it
Compare an arm's edges with its control's before blaming the arm: the platform edges should be equal across arms, and a difference there is chance of timing, not the treatment. Nothing alarms on an edge; the potential-flicker share and the flicker-end alarms watch the apex and these say where to look.
Returns
The Returns card on the XP page (3-dot menu → Returns). Product returns live in NetSuite, not in the bandit database, so the card is a background read: the cohort's purchase lines off the read replica, joined to the refund ledger in the elcap BigQuery export. It is a snapshot as of the run — returns keep posting for weeks after the sale — and the card leads with how much of them it can see yet.
Also shown as Returned, Lines, returned lines
Of the arm's purchase lines that NetSuite knows, the share with a refund posted against the same order and SKU. Counted in lines, one per (order, SKU); a purchase re-sent under the same order id counts once.
How it is computed
The cohort is the stats cohort — first-touch allocation, resellers out, a disabled arm capped at its disabling, a declared winner closing every arm — and its PURCHASE events from each visitor's first exposure on supply the order id and SKU (items[]). A return is a negative Cash Refund or Credit Memo item row in the NetSuite sales stream against that order and SKU; promotional negatives are discounts and are not returns. The line's sale amount is the Sales Order row's (every sale appears as both a Sales Order and a Cash Sale row, so summing them doubles it). A line NetSuite has no row for is unmatched — it is not counted as kept.
How to read it
Read it against the
visible share first: a two-week-old experiment sees a third of its eventual returns, so its rates are early, not final.
vs control is the difference in points with a two-proportion z; the card says "more returns" or "fewer returns" only at |z| ≥ 2 and "within noise" otherwise. Arms of different age are not directly comparable — an arm capped earlier has had longer for its returns to post — and the card says so when their average order age differs by a day or more.
Also shown as of eventual returns, Orders average N d old, mature rate
What share of the returns these orders will eventually see have posted by now, given how old the orders are, on this storefront's own return-lag curve.
How it is computed
The curve is measured on the storefront's website orders placed between 300 and 100 days ago, refunds counted to today: for each day after the sale, the share of the refunds that had posted (refunds later than 90 days are outside the curve and counted only in the mature rate). Each matched line is read off the curve at its own age and the shares are averaged. Mature rate is refunded lines ÷ sold lines on that same order window — the storefront's settled return rate for comparison. Mature read from is the date the youngest order reaches 90% on the curve. The curve is refreshed once a day; a storefront without one gets its rates with the visible share marked unknown.
How to read it
On Backcountry (measured 2026-09-28) about 7% of eventual refunds post within a week, a third within 13 days, 82% within 30 and 93% within 45. Below roughly half visible, a difference between arms is mostly which returns happened to post first — re-run the card from the "mature read" date.
Also shown as multi-size, fit hedging
Of the arm's styles bought (one per order × style), the share bought in two or more SKUs — sizes or colours — in the same order.
How it is computed
Distinct item_variant values per (order, item_id) in the cohort's purchase lines; needs no ledger match, so it is available the day the order is placed. The tooltip carries the difference against control and its z.
How to read it
A leading indicator of fit uncertainty: the second size usually comes back, weeks later. A treatment that raises it (a fit finder that leaves shoppers less sure, say) will show up here long before it shows in the return rate. A single order buying two colours of one style counts too, so read it alongside the category table.
The same return rate per storefront category (the purchase line's item_category), for the fifteen categories with the most cohort lines, one cell per arm.
How it is computed
A cell's z against control is shown only when both cells have at least 30 lines; below that it is withheld as noise rather than shown small.
How to read it
Where a treatment touches one category — a footwear fit finder, a ski-boot size chart — this is the row to read; the all-lines rate dilutes it with everything else the cohort bought.
Investigate and post-mortem
One suite, two faces. A post-mortem reads a run that is over ("may I believe this result"); Investigate reads one still accruing ("keep it running, pause it, or fix something"). The label, the icon and the footnote change; the checks do not. Every number is recomputed from the same services as the readout.
Also shown as Result stands, Read with care, Do not act on this result as it stands, Provisional
The roll-up of every check: pass — the findings are informative, not disqualifying; warn — some findings weaken what the run can be said to show; fail — do not act on it.
How it is computed
Fail dominates. A check that errored (an unreachable model) caps the suite at warn; a skipped check contributes nothing. While checks are still landing the banner is Provisional and can only get stricter, never kinder.
Also shown as Verdict, CVR lift, Realized impact, Sample, Window
Five numbers read off the same verdict as the XP page: the label and its keep hint; the primary challenger's CVR lift and interval; realized impact over challenger visitors; smaller-arm ÷ required visitors with design power; and the exposure window.
The AI half of this check did not run; the row shows its deterministic floor. Five checks are always deterministic (effect-size plausibility, measurement integrity, stability, harm-brake performance, diagnosability); the rest are Claude-backed and degrade to a floor rather than error the suite.
Is the size of the effect believable for what the treatment changed? A cosmetic change (text, attributes, classes, non-layout styles) whose CVR-lift interval sits beyond ±15% is a mechanism (something broke or something is being measured wrong), not a preference. An observed magnitude more than 3× the pre-registered one is "not the same effect".
Also shown as sample ratio mismatch, SRM
Does the control resolve, did every configured variant take traffic, is the split what the allocation says it should be, and does the window cover a full weekly cycle (7 days)? Reseller exclusion above 2% of the cohort is flagged.
How it is computed
Sample-ratio mismatch is a per-arm normal test of observed versus expected share, Bonferroni-corrected across live arms: warn below p = 0.01, fail below p = 0.001. Evaluated only for random and Optimizely allocation — under the bandit an unequal split is the design, so a broken bandit allocation looks exactly like a working one, and the check says so rather than pretending.
Is the effect present from the first visitor (a mechanism) or building over days (a genuine preference)? Day-by-day CVR by first-exposure day, needing 3 usable days of 50+ visitors and a lift beyond 1% before drift is testable.
How it is computed
The newest cohort day is always excluded. Its converters have not all arrived yet, so its CVR is guaranteed to look weak; reporting that as late-onset harm would be a fabricated finding, so the tail is dropped — not adjusted for, dropped.
Did the auto-stop ladder behave — fire when it should have, hold when it should have — and how long did a real harm run before a stop? Reads the persisted monitor snapshot.
Could a harmful result be explained from what was recorded — pre-registration, provenance, tracked items, screenshots? A run that cannot be diagnosed cannot be learned from.
Also shown as Adders → buy, ATC lift, Purchase lift, Device / Share
The device and add-to-cart cuts as a check: a treatment that moves mobile and not desktop is nearly always a layout, viewport or rendering story; purchase moving while add-to-cart does not is a measurement or mix artefact until shown otherwise, because a shopper cannot buy what they never added.
For an experiment with an unsafeJs or loadScript op: the model reads the script for anything that could produce the result mechanically (a broken button, a swallowed click, a beacon fired twice).
Does the result answer the hypothesis and success criteria that were pre-registered? "Nothing was pre-registered, so intent could not be judged" means the check had nothing to compare against.
Also shown as Treatment render, applied N/N ops · ΔCLS
Did the page actually show what the config describes? Each variant is forced in a real browser; the one-liner reports how many ops applied and the layout shift they caused.
What the line and concept had already learned, and whether this result agrees with it or contradicts it.
The model's argument, from every check so far, for why the number moved — reviewed against the live DOM, never describing unobserved state.
The synthesis: keep, pause, fix and re-run, or fork — with the evidence each option rests on.