Distortion controls

Identify how your test could be distorted, by confounding, selection, expectancy, gaming, or peeking, and design against it.

Tonight you start magnesium for sleep, and you already believe in it. Your sister swears by it, you spent 40 dollars on the good brand, and you'll be checking your sleep score each morning with hope. Every one of those facts is a machine for manufacturing a false yes: expectation improves sleep on its own, and checking nightly means you'll declare victory on the first good stretch.

Distortion controls is the move of identifying how your test could be distorted and designing against it before you start. The named distortions include confounding (something else changed too), selection (a skewed sample), expectancy (belief creating the result), gaming (optimizing the number, not the thing), and peeking (stopping the moment the data flatters you). The operation: list which distortions apply, then add one cheap control each. Fix the trial at three weeks before starting, so no peeking. Change nothing else that month, limiting confounds. Log scores without reading them until the end, blunting expectancy. What changes is that a yes at the end is worth believing. Skip elaborate controls when being wrong is nearly free; save them for tests whose answer will steer real money, health, or time.