Reading. Hill, Griffiths & Lim (5th ed.), Probability Primer,
P.1 <80><93>P.2.
A dataset is a sample drawn from a larger population. To learn about the population from the sample, we first need a language for uncertainty
This is the first of three chapters that assemble the probability toolkit we need for inference. Here we set up distributions; the next chapter summarizes a distribution with a single number (expectation); the one after introduces the Normal distribution and the Central Limit Theorem.
A random variable is a variable whose value is unknown until it is observed
Everyday examples are everywhere: the score you will get on the next exam, tomorrow’s value of a stock-market index, the number of games the football team wins next season, the wage of a randomly selected worker. None of these is known in advance, yet each is a number we can reason about.
We write random variables with uppercase letters (\(X, Y, W\)) and the particular values they take with lowercase letters (\(x, y, w\)). So “\(X = x\)” reads: the random variable \(X\) takes the value \(x\).
Think of the population of California adults. Pick one person at random and record their education level. The outcome is not deterministic
But what is a probability? The probability of an outcome is its long-run relative frequency. Saying \(\Prob(\text{bachelor's}) \approx 0.225\) means that across many random draws, about \(22.5\%\) of those drawn hold a bachelor’s degree.
We rarely know the true distribution. Econometrics uses a random sample to make inferences about the underlying distribution.
Every random variable comes with an outcome space \(\mathcal{O}_X\): the set of all values it can take. The single most important distinction in this chapter is whether that set is countable or not.
| Discrete | Continuous |
|---|---|
| Coin flip: \(\{H, T\}\) | Sprint time (s): \([9.5,\,10.5]\) |
| Die roll: \(\{1,2,3,4,5,6\}\) | Income: \([0, \infty)\) |
| Number of doctor visits: \(\{0,1,2,\dots\}\) | Interest rate, GDP, \(\dots\) |
A yes/no answer (“college graduate?”) is a special discrete variable taking only the values \(\{0, 1\}\). We will use these constantly to encode qualitative traits, and they return in force when we study dummy variables.
We describe discrete and continuous variables with different tools
For a discrete random variable, the distribution is captured by the probability mass function.
The pmf of a discrete random variable \(X\) assigns to each possible value \(x\) the probability that \(X\) equals exactly that value: \[ f_X(x) \;=\; \Prob(X = x). \]
\[ \text{(1)}\quad 0 \le f_X(x) \le 1 \qquad\qquad \text{(2)}\quad \sum_{x \in \mathcal{O}_X} f_X(x) = 1 . \]
To get the probability of a set of outcomes \(A\), just add up the masses: \[ \Prob(X \in A) = \sum_{x \in A} f_X(x). \]
Let \(X\) be the result of a fair die roll. Its pmf is \[ f_X(x) = \begin{cases} \tfrac{1}{6} & x \in \{1,2,3,4,5,6\}\\[2pt] 0 & \text{otherwise.} \end{cases} \]
What is the probability of an even roll, \(A = \{2,4,6\}\)? We follow the rule for the probability of a set: \[ \Prob(X \in \{2,4,6\}) = f_X(2)+f_X(4)+f_X(6) = \tfrac{1}{6}+\tfrac{1}{6}+\tfrac{1}{6} = \tfrac{1}{2}. \]
The answer is obvious here
die <- data.frame(x = 1:6, p = 1/6)
ggplot(die, aes(x, p)) +
geom_col(fill = ucla$blue, color = ucla$darkblue, width = 0.6) +
scale_x_continuous(breaks = 1:6) +
scale_y_continuous(
limits = c(0, 0.25),
breaks = c(0, 1/6),
labels = c("0", "1/6")
) +
labs(x = "x", y = expression(f[X](x)))
A discrete distribution is often easiest to read as a table. Consider \(X\) with \[
f_X(1)=0.1,\quad f_X(2)=0.2,\quad f_X(3)=0.3,\quad f_X(4)=0.4 .
\] The probabilities are non-negative and sum to one
pmf_tab <- data.frame(x = c(1, 2, 3, 4, "sum"),
fx = c(0.1, 0.2, 0.3, 0.4, 1.0))
knitr::kable(pmf_tab, col.names = c("$x$", "$f_X(x)$"), align = "cc")| \(x\) | \(f_X(x)\) |
|---|---|
| 1 | 0.1 |
| 2 | 0.2 |
| 3 | 0.3 |
| 4 | 0.4 |
| sum | 1.0 |
d <- data.frame(x = factor(1:4), p = c(0.1, 0.2, 0.3, 0.4))
ggplot(d, aes(x, p)) +
geom_col(fill = ucla$blue, color = ucla$darkblue, width = 0.6) +
scale_y_continuous(limits = c(0, 0.5)) +
labs(x = "x", y = expression(f[X](x)))
We return to this \(X\) below when we build its cdf.
The most important discrete variable in this course takes only two values, \(0\) and \(1\). It is called an indicator (or dummy, or Bernoulli) variable, and it encodes a yes/no trait.
Let \(D = 1\) if a randomly drawn person is a college graduate and \(D = 0\) if not. With \(p = \Prob(D = 1)\), the pmf is \[ f_D(d)= \begin{cases} p & d = 1\\[2pt] 1-p & d = 0\\[2pt] 0 & \text{otherwise.} \end{cases} \] A single number, \(p\), says everything.
Indicators encode qualitative traits
The mean of a \(0/1\) variable is just the proportion of ones: \(\E[D] = p\). We show this in the next chapter
p <- 0.3
bern <- data.frame(d = factor(c(0, 1)), prob = c(1 - p, p))
ggplot(bern, aes(d, prob)) +
geom_col(fill = ucla$blue, color = ucla$darkblue, width = 0.5) +
scale_y_continuous(limits = c(0, 1)) +
labs(x = "d", y = expression(f[D](d)))
For a continuous random variable we cannot use a pmf. Why not?
A continuous variable can take uncountably many values, so the probability of any single exact value is zero: \[ \Prob(X = x) = 0 \quad\text{for every } x. \]
Instead we describe the distribution with a probability density function. Probabilities become areas under the density.
The pdf \(f_X(x)\) of a continuous random variable gives probabilities as areas: \[ \Prob(a \le X \le b) \;=\; \int_a^b f_X(x)\,dx . \]
Notation note. Following HGL we write \(f_X\) for both the discrete pmf and the continuous pdf. Same symbol, different meaning: for a discrete variable \(f_X(x)\) is a probability, while for a continuous variable it is a density
A density \(f_X(x)\) can exceed \(1\)
\[
f_X(x) \ge 0
\qquad\text{and}\qquad
\int_{-\infty}^{\infty} f_X(x)\,dx = 1 .
\] The total area under any density is one
Because single points carry zero probability, endpoints don’t matter: \[ \Prob(a \le X \le b) = \Prob(a < X < b). \]
xs <- seq(-3.5, 3.5, length.out = 400)
dat <- data.frame(x = xs, y = dnorm(xs))
sh <- subset(dat, x >= -1 & x <= 1.5)
ggplot(dat, aes(x, y)) +
geom_area(data = sh, aes(x, y), fill = ucla$blue, alpha = 0.30) +
geom_line(color = ucla$blue, linewidth = 1) +
geom_segment(aes(x = -1, xend = -1, y = 0, yend = dnorm(-1)),
linetype = "dashed", color = ucla$gray) +
geom_segment(aes(x = 1.5, xend = 1.5, y = 0, yend = dnorm(1.5)),
linetype = "dashed", color = ucla$gray) +
annotate("text", x = 0.25, y = 0.16,
label = "P(a <= X <= b)", color = ucla$darkblue, size = 3.4) +
scale_x_continuous(breaks = c(-1, 1.5), labels = c("a", "b")) +
scale_y_continuous(limits = c(0, 0.45)) +
labs(x = "x", y = expression(f[X](x)))
Let \(X\) be uniform on \([0,1]\), with density \[ f_X(x) = \begin{cases} 1 & 0 \le x \le 1\\ 0 & \text{otherwise.} \end{cases} \]
What is \(\Prob(0 \le X \le 0.5)\)? We integrate the density over the interval: \[
\Prob(0 \le X \le 0.5)
= \int_{0}^{0.5} f_X(x)\,dx
= \int_{0}^{0.5} 1 \, dx
= 0.5 .
\] The area is just a rectangle: width \(0.5 \times\) height \(1 = 0.5\). Half the probability sits in the left half of the interval
xs <- seq(-0.4, 1.4, length.out = 300)
dens <- ifelse(xs >= 0 & xs <= 1, 1, 0)
curve_df <- data.frame(x = xs, f = dens)
shade <- data.frame(x = c(0, 0, 0.5, 0.5), y = c(0, 1, 1, 0))
ggplot() +
geom_polygon(data = shade, aes(x, y), fill = ucla$blue, alpha = 0.30) +
geom_line(data = curve_df, aes(x, f), color = ucla$blue, linewidth = 1) +
annotate("text", x = 0.25, y = 0.5, label = "0.5", color = ucla$darkblue) +
scale_x_continuous(breaks = c(0, 0.5, 1)) +
scale_y_continuous(limits = c(0, 1.3), breaks = 1) +
labs(x = "x", y = expression(f[X](x)))
Both discrete and continuous variables share one common summary: the cumulative distribution function, which accumulates probability from \(-\infty\) up to \(x\).
\[ F_X(x) \;=\; \Prob(X \le x). \]
The cdf turns “probability of an interval” into simple subtraction.
\[ \Prob(a < X \le b) \;=\; F_X(b) - F_X(a), \qquad \Prob(X > a) \;=\; 1 - F_X(a). \]
This is exactly how we will read probabilities off statistical tables and software later in the course (Normal and \(t\) probabilities, for instance). We almost never integrate by hand
Take the table from before, \(f_X(1{:}4) = (0.1, 0.2, 0.3, 0.4)\). Accumulating, \[ F_X(1)=0.1,\quad F_X(2)=0.3,\quad F_X(3)=0.6,\quad F_X(4)=1.0 . \]
\[ \Prob(X \le 2) = F_X(2) = 0.1 + 0.2 = 0.3. \] Even a value \(X\) can’t take has a cdf: \(F_X(2.5) = \Prob(X \le 2.5) = 0.3\). And the complement: \(\Prob(X > 2) = 1 - F_X(2) = 0.7\).
The discrete cdf jumps at each possible value, and the size of the jump at \(x\) equals \(f_X(x)\) (Figure 2.6). The closed dots show the value attained at each jump; the open dots show the limit from the left.
seg <- data.frame(
x = c(-0.2, 1, 2, 3, 4),
xend = c(1, 2, 3, 4, 5),
y = c(0, 0.1, 0.3, 0.6, 1.0)
)
closed <- data.frame(x = 1:4, y = c(0.1, 0.3, 0.6, 1.0))
open <- data.frame(x = 1:4, y = c(0.0, 0.1, 0.3, 0.6))
ggplot() +
geom_segment(data = seg, aes(x = x, xend = xend, y = y, yend = y),
color = ucla$blue, linewidth = 1) +
geom_point(data = closed, aes(x, y), color = ucla$blue, size = 2.4) +
geom_point(data = open, aes(x, y), shape = 21, fill = "white",
color = ucla$blue, size = 2.4, stroke = 1) +
scale_x_continuous(breaks = 1:4, limits = c(-0.2, 5)) +
scale_y_continuous(breaks = c(0, 0.1, 0.3, 0.6, 1), limits = c(0, 1.05)) +
labs(x = "x", y = expression(F[X](x)))
For the Uniform\([0,1]\), accumulate the area from the left: \[ F_X(x) = \begin{cases} 0 & x < 0\\ x & 0 \le x \le 1\\ 1 & x > 1. \end{cases} \]
Let’s check the interval rule: \[
\Prob(0.2 < X \le 0.7) = F_X(0.7) - F_X(0.2) = 0.7 - 0.2 = 0.5.
\] A continuous cdf is continuous
xs <- seq(-0.4, 1.4, length.out = 300)
cdf_df <- data.frame(x = xs, F = pmin(pmax(xs, 0), 1))
ggplot(cdf_df, aes(x, F)) +
geom_line(color = ucla$blue, linewidth = 1) +
geom_segment(aes(x = 0.2, xend = 0.2, y = 0, yend = 0.2),
linetype = "dashed", color = ucla$gray) +
geom_segment(aes(x = 0, xend = 0.2, y = 0.2, yend = 0.2),
linetype = "dashed", color = ucla$gray) +
geom_segment(aes(x = 0.7, xend = 0.7, y = 0, yend = 0.7),
linetype = "dashed", color = ucla$gray) +
geom_segment(aes(x = 0, xend = 0.7, y = 0.7, yend = 0.7),
linetype = "dashed", color = ucla$gray) +
scale_x_continuous(breaks = c(0, 0.2, 0.7, 1)) +
scale_y_continuous(breaks = c(0, 0.2, 0.7, 1), limits = c(0, 1.05)) +
labs(x = "x", y = expression(F[X](x)))
A random variable is a numerical outcome that is unknown until observed, described by its distribution. The distinction between discrete and continuous drives which tool we use:
| Discrete (countable \(\mathcal{O}_X\)) | Continuous (interval \(\mathcal{O}_X\)) | |
|---|---|---|
| Describe with | pmf: \(f_X(x) = \Prob(X = x)\) | pdf: area under the curve gives probability |
| Normalization | \(\sum_x f_X(x) = 1\) | \(\int_{-\infty}^{\infty} f_X(x)\,dx = 1\) |
| Probabilities | \(\Prob(X \in A) = \sum_{x \in A} f_X(x)\) | \(\Prob(a \le X \le b) = \int_a^b f_X(x)\,dx\) |
And both share the cdf as a common language: \[ F_X(x) = \Prob(X \le x), \qquad \Prob(a < X \le b) = F_X(b) - F_X(a). \]
Next time: summarizing a distribution with a single number