An Introduction to Probability and Random Variables for Pattern Recognition 1

Axioms

The theory of probability forms the backbone of learning and research in statistical pattern recognition. Probability theory is one of the most amazing fields of mathematics, and it rests on three simple axioms.

1. The probability of an event $A$, written $p(A)$, is greater than or equal to zero.

   $$p(A) \ge 0$$

2. If $A$ and $B$ are mutually exclusive, which means they cannot happen together, then the probability of $A$ or $B$ is the probability of $A$ plus the probability of $B$.

   $$p(A \cup B) = p(A) + p(B) \quad \text{if } A \cap B = \emptyset$$

   The formal statement of this axiom extends the rule to any countable collection of mutually exclusive events.

3. The probability of the entire sample space $\mathcal{S}$ is $1$.

   $$p(\mathcal{S}) = 1$$

These three axioms make probability theory self-checking. No probability is less than $0$, and the probabilities of all the outcomes in a discrete sample space add up to $1$. Also observe that the only arithmetic operator in the axioms is the $+$ (plus) sign. When we begin to combine and apply the axioms, the other arithmetic operations begin to emerge.

Suppose we have a sample space $\mathcal{S}$ with six outcomes, $1$ to $6$, and the probabilities shown in Figure 1 below.

Let us check that this sample space is valid:

1. Is every probability greater than or equal to zero? Yes.
2. Do the probabilities add up to $1$? Yes. $0.1 + 0.1 + 0.3 + 0.1 + 0.2 + 0.2 = 1$.
3. The outcomes are mutually exclusive. So we can find the probability of any event by adding the probabilities of its outcomes. For example:

$$\begin{aligned} p(\text{even}) &= p(\{2, 4, 6\}) \\ &= p(2) + p(4) + p(6) \\ &= 0.1 + 0.1 + 0.2 = 0.4 \end{aligned}$$

Dependence: Conditional Probability

The probability of event $A$ given that event $B$ has already occurred is written $p(A \mid B)$. The vertical bar "$\mid$" reads "given". $p(A \mid B)$ is the probability that both $A$ and $B$ occur, $p(AB)$, divided by the probability of $B$, $p(B)$. The definition holds when $p(B) > 0$.

$$\boxed{p(A \mid B) \triangleq \frac{p(AB)}{p(B)}}$$

This formula is intuitive. Suppose we want the probability of a $3$ given that the outcome is odd, $p(3 \mid \text{odd})$. If we keep only the odd outcomes from the sample space, their probabilities are $0.1$, $0.3$ and $0.2$. These add up to $0.6$, so they no longer form a valid probability distribution. We normalize them by dividing each probability by their sum, $0.6$, as shown in Figure 2. We can then read the probability of a $3$ from the normalized sample space.

The outcome $3$ is odd, so the event "$3$ and odd" is the event "$3$". The result is:

$$p(3 \mid \text{odd}) = \frac{p(3)}{p(\text{odd})} = \frac{0.3}{0.6} = 0.5$$

Rearranging the definition gives the product rule:

$$\boxed{ \begin{gathered} p(A \mid B) = \frac{p(AB)}{p(B)} \\ \text{we then derive that:} \\ p(AB) = p(A \mid B) \, p(B) \end{gathered} } \tag{1}$$

Independence

Two events $A$ and $B$ are independent if knowing that $B$ occurred tells us nothing about whether $A$ occurs. Suppose we are asked for the probability of a $3$ given that it snowed in Lagos, $p(3 \mid \text{snowed in Lagos})$. The answer is simply $p(3)$, because whether it snowed in Lagos has no bearing on the outcome. Formally, $A$ and $B$ are independent when $p(AB) = p(A) \, p(B)$. When $p(B) > 0$, this is the same as saying $p(A \mid B) = p(A)$:

$$\boxed{ \begin{gathered} p(A \mid B) = p(A) \\ \text{and from Equation (1):} \\ p(AB) = p(A \mid B) \, p(B) \\ \therefore \; p(AB) = p(A) \, p(B) \end{gathered} } \tag{2}$$

Total Probability

Suppose a random variable $X$ takes the values $1$ to $6$, and its distribution depends on a second random variable $Y$. $Y$ has two outcomes, $0$ and $1$, with $p(Y = 0) = 0.4$ and $p(Y = 1) = 0.6$. When $Y = 0$, we draw $X$ from the distribution on the left of Figure 3. When $Y = 1$, we draw $X$ from the distribution on the right.

Suppose we are asked:

• What is the probability that $X = 3$ given $Y = 0$, $p(X = 3 \mid Y = 0)$? We read the answer, $0.25$, from the distribution of $X$ when $Y = 0$.
• What is the probability that $X = 4$ given $Y = 1$, $p(X = 4 \mid Y = 1)$? We read the answer, $0.1$, from the distribution of $X$ when $Y = 1$.

Now suppose we are asked for the probability that $X = 3$, $p(X = 3)$. We cannot read this from either distribution because we are not told the value of $Y$. We can still derive it. The events $\{X = 3, Y = 0\}$ and $\{X = 3, Y = 1\}$ are mutually exclusive, and together they make up the event $\{X = 3\}$. The second axiom lets us add their probabilities, and the product rule in Equation (1) gives each of them:

$$\begin{aligned} p(X=3) &= p(X=3, Y=0) + p(X=3, Y=1) \\ &= p(X=3 \mid Y=0) \, p(Y=0) + p(X=3 \mid Y=1) \, p(Y=1) \\ &= (0.25)(0.4) + (0.2)(0.6) \\ &= 0.1 + 0.12 \\ &= 0.22 \end{aligned}$$

In general, when the outcomes of $Y$ are mutually exclusive and cover every possibility, the law of total probability states:

$$p(X = x) = \sum_{y} p(X = x \mid Y = y) \, p(Y = y) \tag{3}$$