SIGNAL gate and diagnostic weight

Audit the quality of the result, then ask how much more expected it was under one explanation than the others.

The A/B test just came back and the new landing page won: 12 percent more signups. The team is ready to ship and celebrate. But the test ran during a promotion week, half the traffic came from one unusual ad campaign, and a 12 percent difference at your traffic volume is within normal wobble. A result this welcome deserves extra suspicion, not less.

SIGNAL gate and diagnostic weight is a two-part move you run on any result before letting it change your mind. First the gate: audit the result's quality on checks that can't be traded against each other, including where it came from, whether its parts are truly independent, how big the leap from data to claim is, whether anyone searched for disconfirming cases, and whether it applies to your situation. A failed check isn't outvoted by strengths elsewhere. Second the weight: ask how much more expected this result was under your favorite explanation than under the rivals. A 12 percent bump during promo week is nearly as expected under "promo noise" as under "better page," so it carries little weight. What changes is that results move you in proportion to what they can bear. Skip the audit only for stakes too small to matter.