Rule of Succession - Generalization To Any Number of Possibilities

Generalization To Any Number of Possibilities

This section gives a heuristic derivation to that given in Probability Theory: The Logic of Science.

The rule of succession has many different intuitive interpretations, and depending on which intuition one uses, the generalisation may be different. Thus, the way to proceed from here is very carefully, and to re-derive the results from first principles, rather than to introduce an intuitively sensible generalisation. The full derivation can be found in Jaynes' book, but it does admit an easier to understand alternative derivation, once the solution is known. Another point to emphasise is that the prior state of knowledge described by the rule of succession is given as an enumeration of the possibilities, with the additional information that it is possible for each category to be observed. This can be equivalently stated as observing each category once prior to gathering the data. To denote that this is the knowledge being used, an I_m will be put as part of the conditions in the probability assignments.

The rule of succession comes from setting a binomial likelihood, and a uniform prior distribution. Thus a straight forward generalisation is just the multivariate extensions of these two distributions: 1)Setting a uniform prior over the initial m categories, and 2) using the multinomial distribution as the likelihood function (which is the multivariate generalisation of the binomial distribution). It can be shown that the uniform distribution is a special case of the Dirichlet distribution with all of its parameters equal to 1 (just as the uniform is Beta(1,1) in the binary case). The Dirichlet distribution is the conjugate prior for the multinomial distribution, which means that the posterior distribution will also be a Dirichlet distribution with different parameters. Let p_i denote the probability that category i will be observed, and let n_i denote the number of times category i (i = 1, ..., m) actually was observed. Then the joint posterior distribution of the probabilities p₁, ..., p_m is given by;

To get the generalised rule of succession, note that the probability of observing category i on the next observation, conditional on the p_i is just p_i, we simply require its expectation. Letting A_i denote the event that the next observation is in category i (i = 1, ..., m), and let n = n₁ + ... + n_m be the total number of observations made. The result, using the properties of the dirichlet distribution is:

This solution reduces to the probability which would be assigned using the principle of indifference before any observations made (i.e. n = 0), consistent with the original rule of succession. It also contains the rule of succession as a special case, when m = 2, as a generalisation should.

Because the propositions or events A_i are mutually exclusive, it is possible to collapse the m categories into 2. Simply add up the A_i probabilities which correspond to "success" to get the probability of success. Supposing that this aggregates c categories as "success" and m-c categories as "failure". Let s denote the sum of the relevant n_i values which have been termed "success". The probability of "success" at the next trial is then:

which is different from the original rule of succession. But note that the original rule of succession is based on I₂, whereas the generalisation is based on I_m. This means that the information contained in I_m is different to that contained in I₂. This indicates that mere knowledge of more than two outcomes which we are sure are possible is relevant information when collapsing these categories down into just two. This illustrates the subtlety in describing the prior information, and why it is important to specify which prior information one is using.