Calibration (Probabilistic)

Probabilistic calibration is consistency between forecast probabilities and observed frequencies over a relevant set of resolved events. It concerns predictions and outcomes together, not certainty about one case.

A binary forecast is calibrated when events assigned a given probability occur at that frequency in the relevant population of forecasts. Calibration is a relation between predictions and outcomes, not a feeling of confidence or a verdict on one prediction. Estimating it requires enough resolved cases and attention to how they were selected.

A calibrated forecast can still be uninformative if it always reports the base rate. Useful forecasting also distinguishes cases and, for distributions, seeks informative concentration consistent with calibration. Apparent overall calibration can conceal errors in subgroups or time periods, so the evaluation must match the intended use.

When to use it

When analyzing data; when building predictive models; when evaluating statistical claims; when quantifying uncertainty.

How it can help

Record forecasts before outcomes, compare frequencies with stated probabilities, account for sample uncertainty, and examine important subgroups. Evaluate informativeness as well as calibration.