How to perform Scheffe’s Test in SAS

Why Scheffe’s Method Excels at Error Control

The decision to use a specific post hoc test is often driven by the need to balance statistical power with the control of errors. Scheffe’s Test is recognized for being the most conservative of the common post hoc procedures. Its critical value is calculated using the F-ratio from the overall ANOVA, which makes it particularly effective at guarding against the inflation of the family-wise error rate across all possible comparisons, including highly complex contrasts. This makes it an ideal choice when the researcher is engaged in exploratory data analysis where specific mean differences were not hypothesized beforehand.

Although this high level of stringency means Scheffe’s Test might possess less statistical power for simple pairwise comparisons compared to alternatives like Tukey’s Honestly Significant Difference (HSD) test, its advantage lies in its general applicability. It allows for the testing of all possible linear combinations of means (contrasts), not just simple pairs, without increasing the chance of a Type I error above the designated alpha level. Therefore, when maximum error protection and flexibility in contrast comparison are paramount, Scheffe’s Test is the procedure of choice for ensuring robust and dependable statistical findings.

Designing the Example Dataset in SAS

To illustrate the practical application of Scheffe’s Test, consider a study where a cognitive psychologist investigates the efficacy of three distinct studying methods (labeled A, B, and C) on final exam performance. A total of 30 students are equally distributed across the three methods, resulting in a balanced design. The scores attained by the students are collected as the dependent variable. The first step in SAS involves structuring this raw data into a usable dataset, naming it my_data.

The data creation utilizes the standard DATA and INPUT steps in SAS programming. The Method variable is defined as a character variable (indicated by the dollar sign, $) since it represents the categorical groupings, while the Score variable is numeric. This meticulous preparation ensures that the subsequent analytical procedures correctly identify the type and role of each variable in the statistical model.

/*create dataset: Three study methods (A, B, C) and their corresponding exam scores.*/
data my_data;
    input Method $ Score;
    datalines;
A 76
A 77
A 77
A 81
A 82
A 82
A 83
A 84
A 85
A 89
B 81
B 82
B 83
B 83
B 83
B 84
B 87
B 90
B 92
B 93
C 77
C 78
C 79
C 88
C 89
C 90
C 91
C 95
C 98
C 98
;
run;

This code block successfully organizes the 30 student scores based on the three study methods, establishing the foundation for testing the hypothesis that differences exist among the methods regarding their effect on exam performance. The subsequent analysis step will then utilize this dataset to run the formal statistical test.

Implementing Scheffe’s Test using PROC ANOVA in SAS

In the SAS system, the Scheffe’s Test is invoked directly within the PROC ANOVA procedure block, specifically through the MEANS statement. This integration simplifies the workflow by executing the omnibus test and the necessary post hoc comparisons sequentially.

The required syntax involves three essential components within the PROC ANOVA statement:

• The CLASS statement identifies the categorical factor (Method).
• The MODEL statement defines the linear relationship, specifying the dependent variable (Score) as a function of the independent variable (Method).
• The MEANS statement is where the post hoc test is requested. We specify the factor (Method) followed by the options SCHEFFE and CLDIFF.

The inclusion of the SCHEFFE keyword instructs SAS to apply Scheffe’s adjustment to all pairwise mean comparisons. The CLDIFF option is particularly useful, requesting confidence intervals for the difference between means. These intervals provide a quantitative measure of the magnitude of the difference, complementing the significance test.

/*perform one-way ANOVA with Scheffe's post-hoc test*/
proc ANOVA data=my_data;
class Method;
model Score = Method;
means Method / scheffe cldiff;
run;

Executing this code generates a comprehensive output that begins with the overall ANOVA summary, crucial for determining if the post hoc results should be interpreted, followed immediately by the detailed pairwise comparison matrix using Scheffe’s adjustment.

Analysis of the Overall ANOVA F-Test

The initial stage of interpreting the SAS output involves reviewing the ANOVA table to confirm that the variation explained by the study Method is statistically significant. If this global test is not significant, the entire post hoc analysis is generally deemed unnecessary, as the observed differences are likely due to random chance rather than the treatment effect.

From the table displayed, we extract the critical values for the Method factor:

• The observed F Value is 3.49.
• The corresponding p-value (Pr > F) is 0.0448.

Based on the standard significance threshold of $alpha = 0.05$, since 0.0448 is less than 0.05, we successfully reject the null hypothesis ($H_0$). This statistical decision confirms the presence of an overall effect: the mean exam scores are not equivalent across all three study methods. Because statistical significance has been established, we now have the justification to proceed to the specific pairwise comparisons generated by Scheffe’s Test.

Interpreting the Scheffe’s Pairwise Comparison Output

The subsequent table in the SAS output presents the controlled pairwise comparisons, allowing us to pinpoint the source of the significant difference found in the omnibus ANOVA. This matrix compares every group mean against every other group mean, using the Scheffe adjustment factor to maintain the overall family-wise error rate.

Significance in this output is typically denoted by asterisks (***) adjacent to the comparison, indicating that the difference between those two groups is statistically reliable even after the conservative Scheffe adjustment. We analyze the results row by row:

1. A vs. C (or C vs. A): The presence of asterisks indicates a statistically significant difference. Study Method C yielded significantly different exam scores than Study Method A.
2. A vs. B (or B vs. A): The lack of asterisks shows no statistically significant difference. The means for methods A and B are statistically similar.
3. B vs. C (or C vs. B): The absence of asterisks confirms no statistically significant difference. The means for methods B and C are statistically similar.

The definitive conclusion is that the significant difference identified by the ANOVA is attributable solely to the contrast between Study Method A and Study Method C. This level of detail, achieved through the robust Scheffe’s Test, is vital for drawing accurate conclusions in experimental research.

Quantifying the Effect Size with Confidence Intervals

Beyond simply declaring whether a difference is significant, the CLDIFF option provides quantitative insight into the magnitude and precision of the observed effect. Focusing on the significant comparison (C vs. A), the output provides both the mean difference and its 95% confidence interval.

The calculated mean difference in exam scores between group C and group A is 6.7. This suggests that, on average, students using Method C scored 6.7 points higher than those using Method A. The associated 95% confidence interval for this true population difference is reported as [0.064, 13.336].

The critical interpretation of this interval confirms the statistical significance because the entire range lies above zero. Since zero is not contained within the interval, we are 95% confident that the true difference between the population means of C and A is positive, reinforcing the rejection of the null hypothesis for this specific pair. Conversely, had the comparison been non-significant (like A vs. B), the confidence interval would span across zero (i.e., contain both positive and negative values), indicating that a true difference of zero is plausible.

Summary of Scheffe’s Application

The execution of Scheffe’s Test in SAS offers a powerful, controlled approach to multi-group comparison following a significant ANOVA. By utilizing the MEANS / SCHEFFE CLDIFF syntax within PROC ANOVA, researchers can efficiently move from detecting a general effect to identifying the specific pairs responsible for that effect, all while rigorously controlling the family-wise error rate. This methodological integrity ensures that statistical conclusions derived from post hoc comparisons are reliable and scientifically sound.

For researchers seeking to conduct advanced statistical comparisons or explore complex contrasts within their data, Scheffe’s method remains an essential technique for robust analysis.

A Guide to Using Post Hoc Tests with ANOVA