How to Easily Distinguish Between Likelihood and Probability

The Fundamental Distinction: Probability vs. Likelihood

The primary difference lies in what is assumed to be known and what is being calculated. When we discuss probability, we are working forward: we assume the statistical model and its underlying parameters are fixed and accurate, and we calculate the chance of observing a specific outcome or set of outcomes. Probability values are strictly constrained between 0 and 1, where 0 signifies impossibility and 1 signifies certainty. It addresses the question: “Given this model, what is the chance of observing this data?”

Conversely, likelihood involves working backward. We observe a set of outcomes (the sample data) and then evaluate how well a specific set of model parameters supports, or “makes likely,” the observed data. Likelihood is a function of the model parameters given the data, quantifying the plausibility of different parameter values after observing the evidence. It is important to note that a likelihood value itself is not a probability; it does not have the same strict bounds or requirements for summation, although comparing relative likelihoods is central to statistical inference.

The relationship can be formalized using conditional notation. If $D$ represents the data and $theta$ represents the parameters, then probability is $P(D|theta)$—the probability of the data given the parameters. Likelihood, on the other hand, is $L(theta|D)$, the likelihood of the parameters given the data. While mathematically related, often through Bayes’ theorem, they address fundamentally different questions regarding the statistical model. When we are calculating the probability of some outcome, we assume the parameters in a model are trustworthy; when we calculate likelihood, we are trying to determine if we can trust those parameters based on the observed data.

A Quick Summary of Definitions

To crystallize the distinction, consider these defining characteristics that separate the two terms in statistical discourse:

• Probability: Refers to the chance that a particular outcome occurs, assuming the true values of the model parameters are known. It predicts data based on a fixed model, moving from parameter to data.
• Likelihood: Refers to the degree to which a specific sample of observed data supports particular hypothesized values of a parameter within a statistical model. It evaluates models based on fixed data, moving from data to parameter inference.

When calculating the probability of a future event, we must trust the underlying assumptions about the population characteristics. For instance, if we assume a population mean ($mu$) is 100, we calculate the probability of drawing a sample mean ($bar{x}$) of 105. When calculating likelihood, however, we use an observed sample mean of 105 and try to determine the plausibility that the true population mean ($mu$) is 100 versus 110, or any other value. This shift in perspective is the critical dividing line.

Example 1: Analyzing Coin Tosses and Fairness

A classic illustration of the difference involves a simple coin toss scenario. Suppose we have a coin that is assumed to be fair. This means the probability of landing on heads ($P_H$) is exactly 0.5. This assumption, $P_H = 0.5$, serves as our known model parameter ($theta$).

If we flip this coin one time, the probability that it will land on heads is calculated directly from our model assumption: $P(text{Heads}|theta=0.5) = 0.5$. If we flip it ten times, the probability of observing exactly five heads is calculated using the binomial distribution, predicated entirely on the parameter $P_H = 0.5$ being correct. When calculating the probability of a coin landing on heads, we simply assume that $P(text{heads}) = 0.5$ on a given toss.

Now, let us shift our focus to likelihood. Suppose we actually flip the coin 100 times and observe the outcome: it lands on heads only 17 times. This observed data ($D$) is now fixed. We use this data to assess the credibility of different parameter values ($theta$). The question becomes: what is the likelihood that the coin is truly fair ($theta=0.5$), given that we only saw 17 heads? When calculating the likelihood, we are trying to determine if the model parameter ($p=0.5$) is actually correctly specified.

Given the data of 17 heads out of 100 tosses, the likelihood that the coin is truly fair ($theta=0.5$) is quantitatively quite low. If the coin were genuinely fair, we would expect an outcome much closer to 50 heads. The observed sample makes us highly suspicious that the true probability of the coin landing on heads on a given toss is actually $p=0.5$. Thus, while probability predicts the outcome given the fairness, likelihood assesses the fairness given the outcome.

Example 2: Probability and Likelihood in Spinners

Consider a hypothetical spinner split into thirds with three colors on it: Red, Green, and Blue. Suppose we assume that it is equally likely for the spinner to land on any of the three colors. Our initial assumption (the parameter $theta$) is a uniform distribution, meaning the probability of landing on any one color is $1/3$.

If we spin it one time, the probability that it lands on red is $P(text{Red}|theta) = 1/3$. This calculation is straightforward because we assume the model parameters (equal probability for all sections) are accurate and reliable. When calculating the probability of the spinner landing on red, we simply assume that $P(text{red}) = 1/3$ on a given spin.

Now suppose we spin it 100 times and record the results: Red appears 2 times, Green appears 90 times, and Blue appears 8 times. The data is now fixed. We must use this data to calculate the likelihood of our initial parameter assumption ($theta$: Red=1/3, Green=1/3, Blue=1/3). We would say that the likelihood that the spinner is actually equally likely to land on each color is very low.

The observed results, specifically 90 out of 100 spins landing on Green, make the initial assumption of equal probability highly suspicious. The results of the 100 spins make us believe that the model parameter is incorrectly specified. Consequently, the data confers an extremely low likelihood on the hypothesis that the spinner is fair. This example demonstrates that likelihood is the metric used to judge how well a theoretical model fits real-world observations.

Example 3: Assessing Claims in Gambling Scenarios

Imagine a scenario where a casino claims that the probability of winning money on a certain slot machine is 40% for each turn. This claim represents our model parameter $theta$, where $P(text{Winning}) = 0.40$.

If we take one turn, the probability that we will win money is 0.40. We trust the parameter $theta=0.40$ and use it to calculate the probability of the outcome. This is a predictive measure based on the stated odds.

Now, suppose we take 100 turns and win 42 times. The observed data ($D$) is 42 wins out of 100 trials. We calculate the likelihood of the casino’s claim ($theta=0.40$) given this data. When calculating the likelihood, we are trying to determine if the model parameter $P(text{winning}) = 0.40$ is actually correctly specified.

Since the sample proportion of wins (0.42) is very close to the claimed probability (0.40), we would conclude that the likelihood that the probability of winning is truly 40% seems quite fair. Winning 42 times out of 100 makes us believe that a probability of winning 40% of the time seems reasonable. If the outcome were drastically different, say 10 wins out of 100, the likelihood assigned to the $P=0.40$ parameter would drop severely, leading us to reject the casino’s claim.

The Computational Power of Maximum Likelihood

The practical application of likelihood is perhaps best demonstrated through its role in parameter estimation, particularly the widely used technique known as Maximum Likelihood Estimation (MLE). MLE is a statistical method designed to estimate the parameters of a model by finding the parameter values that maximize the probability of obtaining the specific observed data set.

Stated simply, MLE searches for the parameter value that makes the observed data most likely to have occurred. For example, returning to the coin toss where 17 heads were observed in 100 flips, the MLE for the true probability of heads is $p=0.17$. This estimate is the one that maximizes the likelihood function for the given sample, providing the most justifiable parameter estimate based on empirical evidence.

Unlike some other estimation methods, MLE possesses desirable statistical properties, such as consistency and efficiency, especially for large sample sizes. This robustness makes the likelihood function a cornerstone of modern statistical inference, powering everything from linear regression models to complex machine learning algorithms.

Summary of Key Contrasts

To ensure a clear understanding of these concepts, here is a concise comparison focusing on their statistical function:

1. Direction of Inquiry: Probability is deductive, predicting data from a known parameter set. Likelihood is inductive, inferring parameter plausibility from fixed data.
2. The Fixed Quantity: In probability, the model parameter is fixed and known. In likelihood, the observed data is fixed and known.
3. Output Value: Probability results in a value between 0 and 1, representing a chance. Likelihood results in a relative value (often unconstrained) used for comparison and optimization.
4. Statistical Goal: Probability is used for prediction and characterizing variation given a model. Likelihood is used for statistical inference, parameter estimation, and comparing competing hypotheses about the model.

Conclusion: The Importance of Precision in Statistical Language

Avoiding the colloquial confusion between probability and likelihood is essential for anyone engaged in serious statistical analysis or data modeling. While probability gives us the mathematical calculation of chance based on fixed assumptions, likelihood provides the indispensable tool for testing and validating those underlying assumptions using real-world evidence. Mastery of this distinction ensures that statistical conclusions are sound, transparent, and correctly framed within the context of scientific inquiry.

The following tutorials provide additional detailed information about probability and related statistical concepts: