This page will demonstrate the use of the anovalator command (search anovalator) to
compute simple main effects for both a two-factor model and a three-factor model. We will be using
anovalator with the anova command on this page but anovalator works equally well
with regress, xtmixed and many other estimation commands.

Example 1: Two-factor model

For our first example we will use the hsbanova dataset.

use https://stats.idre.ucla.edu/stat/data/hsbanova, clear

anova write grp##female

                           Number of obs =     200     R-squared     =  0.2872
                           Root MSE      = 8.14699     Adj R-squared =  0.2612

                  Source |  Partial SS    df       MS           F     Prob > F
              -----------+----------------------------------------------------
                   Model |  5135.17494     7   733.59642      11.05     0.0000
                         |
                     grp |  3641.68311     3  1213.89437      18.29     0.0000
                  female |  984.377328     1  984.377328      14.83     0.0002
              grp#female |  575.513416     3  191.837805       2.89     0.0367
                         |
                Residual |  12743.7001   192  66.3734378   
              -----------+----------------------------------------------------
                   Total |   17878.875   199   89.843593

You will note that the grp by female interaction is statistically significant.
We will show two ways of using anovalator to test the effect of grp for each
level of female. The first method uses the simple option in anovalator.
This approach will only work with two-factor models. We will also include the fratio
option in all of our examples since the disturbances are assumed to be normally distributed
in our anova model.

anovalator grp female, simple fratio

anovalator test of simple main effects for grp at(female=0)
chi2(3) = 46.001924   p-value = 5.666e-10
scaled as F-ratio = 15.333975

anovalator test of simple main effects for grp at(female=1)
chi2(3) = 13.644845   p-value = .00343069
scaled as F-ratio = 4.5482816

We can obtain the same results using anovalator with the maineffect
(abbreviated main) and at options.

anovalator grp, main fratio at(female=0)

anovalator main-effect for grp at(female=0)
chi2(3) = 46.001924   p-value = 5.666e-10
scaled as F-ratio = 15.333975

anovalator grp, main fratio at(female=1)

anovalator main-effect for grp at(female=1)
chi2(3) = 13.644845   p-value = .00343069
scaled as F-ratio = 4.5482816

Since tests of simple main effects are a type of post-hoc comparison we need to use an adjusted
critical value to assess statistical significance. We will use the smecriticalvalue
command (search smecriticalvalue) for this purpose. The smecriticalvalue
command needs four pieces of information: n – the number of test performed; df1 –
the degrees of freedom for each test; df2 – the degrees of freedon for the error term in
the anova model; and dfmodel – the total degrees of freedom for all the terms in the model.

smecriticalvalue, n(2) df1(3) df2(192) dfmodel(7)

  number of tests: 2
     numerator df: 3
   denominator df: 192
original model df: 7

Critical value of F for alpha = .05 using ...
------------------------------------------------
Dunn's procedure              = 3.3028802
Marascuilo & Levin            = 3.6129331
per family error rate         = 3.1847981
simultaneous test procedure   = 3.6365283

By any of the four criteria above both tests of simple main effects are statistically significant.

Example 2: Three-factor model

Next we will run a three-factor model using the threeway dataset.

use https://stats.idre.ucla.edu/stat/data/threeway, clear

anova y a##b##c

                           Number of obs =      24     R-squared     =  0.9689
                           Root MSE      =  1.1547     Adj R-squared =  0.9403

                  Source |  Partial SS    df       MS           F     Prob > F
              -----------+----------------------------------------------------
                   Model |  497.833333    11  45.2575758      33.94     0.0000
                         |
                       a |         150     1         150     112.50     0.0000
                       b |  .666666667     1  .666666667       0.50     0.4930
                     a#b |  160.166667     1  160.166667     120.13     0.0000
                       c |  127.583333     2  63.7916667      47.84     0.0000
                     a#c |       18.25     2       9.125       6.84     0.0104
                     b#c |  22.5833333     2  11.2916667       8.47     0.0051
                   a#b#c |  18.5833333     2  9.29166667       6.97     0.0098
                         |
                Residual |          16    12  1.33333333   
              -----------+----------------------------------------------------
                   Total |  513.833333    23  22.3405797

The a#b#c threeway interaction is statistically significant. Prior research has
suggested that we look at the b#c interaction for each level of a. Because
this is a three-factor design we cannot use the simple option so we will use a variation
of the second method shown above, this time using the twoway (abbreviated two)
option.

anovalator grp female, two fratio at(a=1)

anovalator two-way interaction for b#c at(a=1)
chi2(2) = 30.5   p-value = 2.382e-07
scaled as F-ratio = 15.25

anovalator b c, two fratio at(a=2)

anovalator two-way interaction for b#c at(a=2)
chi2(2) = .375   p-value = .82902912
scaled as F-ratio = .1875

Since the b#c at a=1 is significant we will follow this analysis up
by looking at c for each level of b while keeping a at level one.

anovalator c, main fratio at(a=1 b=1)

anovalator main-effect for c at(a=1 b=1)
chi2(2) = 48   p-value = 3.775e-11
scaled as F-ratio = 24

anovalator c, main fratio at(a=1 b=2)

anovalator main-effect for c at(a=1 b=2)
chi2(2) = 1   p-value = .60653066
scaled as F-ratio = .5

We will use the smecriticalvalue command once again, this including all four
tests since they have the same degrees of freedom.

smecriticalvalue, n(4) df1(2) df2(12) dfmodel(11)

  number of tests: 4
     numerator df: 2
   denominator df: 12
original model df: 11

Critical value of F for alpha = .05 using ...
------------------------------------------------
Dunn's procedure              = 6.2753765
Marascuilo & Levin            = 7.1335873
per family error rate         = 6.4546898
simultaneous test procedure   = 10.245969

There really wasn’t much mystery here. Clearly, the F-ratios of .1875 and .5 cannot be significant.
While the F-ratios of 15.25 and 24 both exceed the most stringent criteria, the
simultaneous test procedure.