A one-way ANOVA is used to determine whether or not there is a statistically significant difference between the means of three or more independent groups.

If the overall p-value from the ANOVA table is less than some significance level, then we have sufficient evidence to say that at least one of the means of the groups is different from the others.

However, this doesn’t tell us which groups are different from each other. It simply tells us that not all of the group means are equal.

In order to find out exactly which groups are different from each other, we must conduct a post-hoc test that is capable of controlling thefamily-wise error rate.

One of the most commonly used post hoc tests is Scheffe’s test.

This tutorial explains how to perform Scheffe’s test in R.

Example: Scheffe’s Test in R

Suppose a teacher wants to know whether or not three different studying techniques lead to different exam scores among students. To test this, she randomly assigns 10 students to use each studying technique and records their exam scores.

We can use the following steps in R to fit a one-way ANOVA to test for differences in mean exam scores among the three groups and use Scheffe’s test to determine exactly which groups are different.

Step 1: Create the dataset.

The following code shows how to create a dataset that contains exam scores for all 30 students:

#create data frame
data <- data.frame(technique = rep(c("tech1", "tech2", "tech3"), each = 10),
                   score = c(76, 77, 77, 81, 82, 82, 83, 84, 85, 89,
                             81, 82, 83, 83, 83, 84, 87, 90, 92, 93,
                             77, 78, 79, 88, 89, 90, 91, 95, 95, 98))

#view first six rows of data frame
head(data)

  technique score
1     tech1    76
2     tech1    77
3     tech1    77
4     tech1    81
5     tech1    82
6     tech1    82

Step 2: Visualize the exam scores for each group.

The following code shows how to produce boxplots to visualize the distribution of exam scores for each group:

boxplot(score ~ technique,
        data = data,
        main = "Exam Scores by Studying Technique",
        xlab = "Studying Technique",
        ylab = "Exam Scores",
        col = "steelblue",
        border = "black")

Step 3: Perform a one-way ANOVA.

#fit the one-way ANOVA model
model <- aov(score ~ technique, data = data)

#view model output
summary(model)

            Df Sum Sq Mean Sq F value Pr(>F)  
technique    2  211.5  105.73   3.415 0.0476 *
Residuals   27  836.0   30.96                 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Since the overall p-value (0.0476) is less than .05, this is an indication that each group does not have the same average exam score.

Next, we will perform Scheffe’s test to determine which groups are different.

Step 4: Perform Scheffe’s Test.

To perform Scheffe’s test, we’ll use the ScheffeTest() function from the DescTools package.

The following code shows how to use this function for our example:

#load DescTools package
library(DescTools)

#perform Scheffe's test
ScheffeTest(model)

  Posthoc multiple comparisons of means : Scheffe Test 
    95% family-wise confidence level

$technique
            diff      lwr.ci    upr.ci   pval    
tech2-tech1  4.2 -2.24527202 10.645272 0.2582    
tech3-tech1  6.4 -0.04527202 12.845272 0.0519 .  
tech3-tech2  2.2 -4.24527202  8.645272 0.6803    

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The way to interpret the output is as follows:

• The mean difference in exam scores between technique 2 and technique 1 is 4.2. The corresponding p-value for the mean difference is .2582.
• The mean difference in exam scores between technique 3 and technique 1 is 6.4. The corresponding p-value for the mean difference is .0519.
• The mean difference in exam scores between technique 3 and technique 2 is 2.2. The corresponding p-value for the mean difference is .6803.

Depending on the significance level we decide to use, the only two groups that seem to be statistically significantly different are technique 3 and technique 1.

How to Conduct a One-Way ANOVA in R
How to Perform Tukey’s Test in R
How to Perform a Bonferroni Correction in R