What is difference between P-Value and Alpha?

---

Two terms that students often get confused in statistics are p-value and alpha. 

Both terms are used in , which are formal statistical tests we use to reject or fail to reject some hypothesis.

For example, suppose we hypothesize that a new pill reduces blood pressure in patients more than the current standard pill.

To test this, we can conduct a hypothesis test where we define the following null and alternative hypotheses:

Null hypothesis: There is no difference between the new pill and the standard pill.

Alternative hypothesis: There is a difference between the new pill and the standard pill.

If we assume the null hypothesis is true, the p-value of the test tells us the probability of obtaining an effect at least as large as the one we actually observed in the sample data. 

For example, suppose we find that the p-value of the hypothesis test is 0.02.

Here’s how to interpret this p-value: If there truly was no difference between the new pill and the standard pill, then 2% of the times that we perform this hypothesis test we would obtain the effect observed in the sample data, or larger, simply due to random sample error.

This tells us that obtaining the sample data that we actually did would be pretty rare if indeed there was no difference between the new pill and the standard pill.

Thus, we would be inclined to reject the statement in the null hypothesis and conclude that there is a difference between the new pill and the standard pill.

But what threshold should we use to determine if our p-value is low enough to reject the null hypothesis?

This is where alpha comes in!

The Alpha Level

The alpha level of a hypothesis test is the threshold we use to determine whether or not our p-value is low enough to reject the null hypothesis. It is often set at 0.05 but it is sometimes set as low as 0.01 or as high as 0.10.

For example, if we set the alpha level of a hypothesis test at 0.05 and we get a p-value of 0.02, then we would reject the null hypothesis since the p-value is less than the alpha level. Thus, we would conclude that we have sufficient evidence to say the alternative hypothesis is true.

It’s important to note that the alpha level also defines the probability of incorrectly rejecting a true null hypothesis.

If we set the alpha level of a hypothesis test at 0.05 then this means that if we repeated the process of performing the hypothesis test many times, we would expect to incorrectly reject the null hypothesis in about 5% of the tests.

How to Choose the Alpha Level

As mentioned earlier, the most common choice for the alpha level of a hypothesis test is 0.05. However, in some situations where there are serious consequences for making incorrect conclusions, we may set the alpha level to be even lower, perhaps at 0.01.

For example, in the medical field it’s common for researchers to set the alpha level at 0.01 because they want to be highly confident that the results of a hypothesis test are reliable.

Conversely, in fields like marketing it may be more common to set the alpha level at a higher level like 0.10 because the consequences for being wrong aren’t life or death.

It’s worth noting that increasing the alpha level of a test will increase the chances of finding a significance test result, but it also increases the chances that we incorrectly reject a true null hypothesis.

Summary:

Here’s what we learned in this article:

1. A p-value tells us the probability of obtaining an effect at least as large as the one we actually observed in the sample data. 

2. An alpha level is the probability of incorrectly rejecting a true null hypothesis.

3. If the p-value of a hypothesis test is less than the alpha level, then we can reject the null hypothesis.

4. Increasing the alpha level of a test increases the chances that we can find a significant test result, but it also increases the chances that we incorrectly reject a true null hypothesis.