Chi-square difference tests are frequently used to test differences between
nested models in confirmatory factor analysis, path analysis and structural equation modeling. Nested models are
two models (or more if one is fitting a series of models) that are identical except that one
of the models constrains some of the parameters (the null model) and one does not have those constraints
(the alternative model). Examples of this include the introduction of a set of
dichotomous predictors representing a single nominal (categorical) variable to the model, or a test for differences across groups in a multiple
group model. Typically a chi-square difference test involves calculating the difference between the chi-square
statistic for the null and alternative models, the resulting statistic is distributed chi-square
with degrees of freedom equal to the difference in the degrees of freedom between the two models.
However, when a model is run in Mplus using the MLM or MLR estimators, the following warning
message is displayed, warning the user that the standard chi-square difference
test is not valid:

*   The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used
    for chi-square difference tests.  MLM, MLR and WLSM chi-square difference
    testing is described in the Mplus Technical Appendices at www.statmodel.com.
    See chi-square difference testing in the index of the Mplus User's Guide.

This page shows how to calculate the correct chi-square difference test statistic for
models estimated with the MLM or MLR estimator as described on the Mplus
website (see http://www.statmodel.com/chidiff.shtml
for additional information). For both MLM and MLR the Satorra-Bentler scaled
chi-square difference test can be used. An additional test is available to test
for differences in nested models that use the MLR estimator; this test is based on the log-likelihoods
from nested models. Information on how to calculate the Satorra-Benter scaled
chi-square test appears first, followed by information on using the
log-likelihood to calculate a difference test. Each section includes both a
discussion of the formulae and an example.

Cautions

While the above message appears for a number of estimators, the
procedure described below is for use with the MLM and MLR estimators only. If
the model uses the MLMV or WLSMV estimator, the difftest command can be
used to test for differences across nested models. A Wald test can also be used to test
nested models using the model test command. See the Mplus manual for
more information on these commands. If you are unsure of what estimator is being used in your model, you can find the estimator listed towards the top of the output file.

Also keep in mind that this type of test is only valid if the models are nested; that is, the models
must be the same except that one of the models includes additional constraints
on the parameters.

The Satorra-Bentler scaled chi-square difference test

In order to calculate the Satorra-Bentler scaled chi-square difference test, we will
need a number of pieces of information.  Below is a list of the information needed, along
with the symbol (i.e., letter and number) used to represent each value.

c0 = scaling correction factor for the null model.
c1 = scaling correction factor for the alternative model.

d0 = degrees of freedom for the null model.
d1 = degrees of freedom for the alternative model.

SB0 = Satorra-Bentler scaled chi-square value for the null model.
SB1 = Satorra-Bentler scaled chi-square value for the alternative model.

In order to calculate the test statistic, T, we first need to calculate
the value cd:

cd = (d0 * c0 - d1 * c1)/(d0 - d1)

Once we have calculated cd, we can compute:

T = (SB0 * c0 - SB1 * c1)/cd

T is distributed chi-square with degrees of freedom:

df = d0 - d1

Example

Below are two Mplus input files. On the left is the null model, which constrains
the paths from x2 to f1 and from x3 to f1 to be zero (using x2@0 x3@0 respectively). On the
right is the
alternative model which estimates the regression path from x2 to f1
and from x3 to f1.

Once we have run the model, we can find all the information we need to compute the test
statistic in the section of the output labeled “Chi-Square Test of Model Fit.”
This section of the output appears below (all other output is omitted). Output for the null model appears on the left,
and output for
the alternative model appears on the right. The information we need appears in
bold. Below the output, each of the values highlighted appears on the
appropriate line. If your output does not include the scaling correction factors
(c0 and c1), instructions for calculating them from other output appears below.

Chi-Square Test of Model Fit

   Value                            178.097*
   Degrees of Freedom                     8
   P-Value                           0.0000
   Scaling Correction Factor          1.018
   for MLR

Chi-Square Test of Model Fit

   Value                             35.122*
   Degrees of Freedom                     6
   P-Value                           0.0000
   Scaling Correction Factor          0.958
     for MLR

c0 = 1.018   (scaling correction factor for the null model)
c1 = 0.958   (scaling correction factor for the alternative model)

d0 = 8       (degrees of freedom for the null model)
d1 = 6       (degrees of freedom for the alternative model)

SB0 = 178.097 (the Satorra-Bentler adjusted chi-square value for the null model)
SB1 = 35.122 (the Satorra-Bentler adjusted chi-square value for the alternative model)

Once we have identified and noted all the necessary values in our output,
the test statistic (T) is computed in two steps.

cd = (d0 * c0 - d1 * c1)/(d0 - d1)
   = (8 * 1.018 - 6 * 0.958)/(8 - 6) = 1.198

T = (SB0 * c0 - SB1 * c1)/cd
  = (178.097 * 1.018 - 35.122 * 0.958) / 1.198 = 123.25198

The test statistic T is distributed chi-square with df = d0-d1. We can
look up the p-value for a chi-square statistic of 123.25, with two degrees of
freedom using a table or some other method (chi2(2) = 123.25, p < 0.01).

Calculating the scaling correction factor

If your output does not include the scaling correction factor, you can calculate this
value from the unadjusted chi-square and the Satorra-Bentler scaled chi-square statistic.
So, we are going to pretend that we have output that contains only the unadjusted chi-square,
the Satorra-Bentler scaled chi-square statistic and the degrees of freedom.

T0 = Unadjusted chi-square value for the null model.
T1 = Unadjusted chi-square value for the alternative model.

SB0 = The Satorra-Bentler scaled chi-square for the null model.
SB1 = The Satorra-Bentler scaled chi-square for the alternative model.

Given those values, the scaling correction factors (c0 and c1) can be calculated
as shown below. c0 is the scaling correction factor for the null model, and c1
is the scaling correction factor for the alternative model.

c0 = T0/SB0
c1 = T1/SB1

For example, taking the appropriate values from the output:

T0 = 181.303
T1 = 33.647 

SB0 = 178.097
SB1 = 35.122

The scaling correction factors can be calculated:

c0 = 181.303/178.097  = 1.018 
c1 = 33.647/35.122 = 0.958

A test using the log-likelihood

For the MLR estimator there is an additional test for nested models.
This test compares the log-likelihoods for the null and alternative models rather
than the chi-square values. Below is a list of the
information needed, along with the symbol (i.e., letter and number) used to represent each value.

L0 = log-likelihood for the null model.
L1 = log-likelihood for the alternative model.

c0 = scaling correction factor for the null model.
c1 = scaling correction factor for the alternative model.

p0 = number of parameters estimated in the null model.
p1 = number of parameters estimated in the alternative model.

From this information the tests statistic TRd can be calculated (note that cd is calculated
first then used to calculate TRd), along with the degrees of freedom:

cd = (p0*c0-p1*c1)/(p0-p1)
TRd = -2*(L0-L1)/cd
df = p1-p0

Example

Below is an example of output from two nested models. The information
necessary to calculate this test statistic appears in the sections of output
labeled “Loglikelihood” and the section immediately after it labeled
“Information Criteria” (all other output has been omitted). The information
needed for the calculations appears in bold. Note that the log likelihoods and
scaling correction factors identified as H0 should be used, not the values
labeled H1. Below the output, each of the values highlighted appears on the
appropriate line.

Loglikelihood

      H0 Value                       -3063.145    
      H0 Scaling Correction Factor       0.942
        for MLR
      H1 Value                       -2972.450
      H1 Scaling Correction Factor       0.976
        for MLR

Information Criteria

      Number of Free Parameters             10   
      Akaike (AIC)                    6146.290
      Bayesian (BIC)                  6179.273
      Sample-Size Adjusted BIC        6147.592
        (n* = (n + 2) / 24)

Loglikelihood

      H0 Value                       -2989.266    
      H0 Scaling Correction Factor       0.985  
        for MLR
      H1 Value                       -2972.450  
      H1 Scaling Correction Factor       0.976  
        for MLR

Information Criteria

      Number of Free Parameters             12
      Akaike (AIC)                    6002.533
      Bayesian (BIC)                  6042.112
      Sample-Size Adjusted BIC        6004.095
        (n* = (n + 2) / 24)