Mutual Information (Shared Knowledge Between Variables)

Mutual information measures statistical dependence between variables, including nonlinear dependence. It is symmetric and is zero at independence when well-defined; estimation from finite data introduces uncertainty.

Mutual information measures statistical dependence by comparing a joint distribution with the distribution that would apply if its variables were independent. In familiar discrete settings it can be written as the reduction in uncertainty about one variable after observing the other. It is symmetric, nonnegative, and zero exactly at independence when the quantity is well-defined; it does not identify a causal direction.

Its practical value depends on estimation and the question being asked. Nonlinear dependence can exist without linear correlation, while a large estimated mutual information can reflect sampling error, leakage, or an irrelevant relationship. Variables can also be informative jointly even when each is uninformative alone. Use suitable estimation and validation rather than treating a feature score as proof of causal importance or future predictive performance.

When to use it

When studying dependence, evaluating information in features, or comparing what additional observations contribute to a prediction.

How it can help

Use an appropriate estimator, inspect the data, guard against leakage, and validate incremental predictive value. Consider interactions rather than screening every variable only in isolation.