P-Hacking

P-hacking refers to analysis or data-collection flexibility used to obtain and selectively report favorable statistical significance without properly accounting for the search. It can arise deliberately or inadvertently.

Selective analysis can make a nominal p-value misleading when it is reported as though one fixed test had been performed. Trying outcomes, subgroups, exclusions, stopping times, or specifications and highlighting only a favorable result changes the selection process. This can happen through motivated or unrecognized flexibility; identifying it does not require proving deliberate misconduct.

Exploration is useful when its role and scope are disclosed. Confirmatory claims need an analysis and sampling procedure whose uncertainty calculation accounts for the choices made. Preregistration, complete reporting, appropriate multiplicity methods, and new-data confirmation address different parts of the problem. Effect sizes can also be inflated by selection, so replacing p-values with them is not a complete remedy.

When to use it

When analyzing data; when building predictive models; when evaluating statistical claims; when quantifying uncertainty.

How it can help

Disclose outcomes, exclusions, specifications, and stopping decisions; distinguish exploration from confirmation; account for selection; and evaluate promising findings on suitable new data.