Discrimination Information
The Kullback–Leibler divergence DKL( p(x|H1) || p(x|H0) ) can also be interpreted as the expected discrimination information for H1 over H0: the mean information per sample for discriminating in favor of a hypothesis H1 against a hypothesis H0, when hypothesis H1 is true. Another name for this quantity, given to it by I.J. Good, is the expected weight of evidence for H1 over H0 to be expected from each sample.
The expected weight of evidence for H1 over H0 is not the same as the information gain expected per sample about the probability distribution p(H) of the hypotheses,
- DKL( p(x|H1) || p(x|H0) ) IG = DKL( p(H|x) || p(H|I) ).
Either of the two quantities can be used as a utility function in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.
On the entropy scale of information gain there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the logit scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the Riemann hypothesis is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of loss function for uncertainty are both useful, according to how well each reflects the particular circumstances of the problem in question.
Read more about this topic: Kullback–Leibler Divergence
Famous quotes containing the word information:
“But while ignorance can make you insensitive, familiarity can also numb. Entering the second half-century of an information age, our cumulative knowledge has changed the level of what appalls, what stuns, what shocks.”
—Anna Quindlen (b. 1952)