Modification Indices in umx

Introduction

Modification indices (MIs) flag single parameters whose addition or removal would most improve model fit (if freed and the model re-estimated). They are useful for exploring where a misspecified model struggles, but they are not a substitute for theory-driven model comparison.

umxMI() wraps OpenMx::mxMI() and prints the largest indices (by default those above \(\chi^2_1(0.01) = 6.63\)). By default full = FALSE (only the tested parameter moves — fast). Set full = TRUE to allow all free parameters to re-adjust (slower).

This vignette uses the classic Holzinger & Swineford (1939) data (HS.ability.data in OpenMx): fit a deliberately poor one-factor model (the worst of the usual teaching sequence), run one umxMI() call, then show a better three-factor alternative with umxCompare() (no second MI pass).

Data

data("HS.ability.data", package = "OpenMx")

manifests = c("visual", "cubes", "flags", "paragrap", "sentence", "wordm",
  "addition", "counting", "straight")

# Scale for stable optimization (recommended for this dataset in umx)
df = umx_scale(HS.ability.data)

One-factor model (worst fit → one MI call)

m1 = umxRAM("Holzinger_1f", data = df,
  umxPath("g", to = manifests),
  umxPath(v.m. = manifests),
  umxPath(v1m0 = "g")
)
umxSummary(m1, refModels = FALSE)
#> 
#> 
#> Table: Parameter loadings for model 'Holzinger_1f'
#> 
#> |   |name                   |Estimate |SE   |CI                |type            |
#> |:--|:----------------------|:--------|:----|:-----------------|:---------------|
#> |1  |g_to_straight          |0.31     |0.06 |0.31 [0.19, 0.42] |Factor loading  |
#> |2  |g_to_counting          |0.2      |0.06 |0.2 [0.08, 0.32]  |Factor loading  |
#> |3  |g_to_addition          |0.18     |0.06 |0.18 [0.06, 0.3]  |Factor loading  |
#> |4  |g_to_wordm             |0.84     |0.05 |0.84 [0.74, 0.93] |Factor loading  |
#> |5  |g_to_sentence          |0.84     |0.05 |0.84 [0.74, 0.94] |Factor loading  |
#> |6  |g_to_paragrap          |0.85     |0.05 |0.85 [0.75, 0.94] |Factor loading  |
#> |7  |g_to_flags             |0.22     |0.06 |0.22 [0.1, 0.34]  |Factor loading  |
#> |8  |g_to_cubes             |0.22     |0.06 |0.22 [0.1, 0.34]  |Factor loading  |
#> |9  |g_to_visual            |0.44     |0.06 |0.44 [0.32, 0.55] |Factor loading  |
#> |19 |g_with_g               |1        |0    |1 [1, 1]          |Factor Variance |
#> |20 |one_to_straight        |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |21 |one_to_counting        |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |22 |one_to_addition        |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |23 |one_to_wordm           |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |24 |one_to_sentence        |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |25 |one_to_paragrap        |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |26 |one_to_flags           |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |27 |one_to_cubes           |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |28 |one_to_visual          |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |10 |straight_with_straight |0.9      |0.07 |0.9 [0.76, 1.05]  |Residual        |
#> |11 |counting_with_counting |0.96     |0.08 |0.96 [0.8, 1.11]  |Residual        |
#> |12 |addition_with_addition |0.96     |0.08 |0.96 [0.81, 1.12] |Residual        |
#> |13 |wordm_with_wordm       |0.3      |0.04 |0.3 [0.23, 0.37]  |Residual        |
#> |14 |sentence_with_sentence |0.29     |0.04 |0.29 [0.22, 0.36] |Residual        |
#> |15 |paragrap_with_paragrap |0.28     |0.04 |0.28 [0.21, 0.35] |Residual        |
#> |16 |flags_with_flags       |0.95     |0.08 |0.95 [0.79, 1.1]  |Residual        |
#> |17 |cubes_with_cubes       |0.95     |0.08 |0.95 [0.8, 1.1]   |Residual        |
#> |18 |visual_with_visual     |0.81     |0.07 |0.81 [0.67, 0.94] |Residual        |

Fit is poor (as expected for a single factor on these data). What does umxMI() suggest?

umxMI(m1)
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings,  : 
#>   A matrix has non-zero diagonal
#> addition_with_counting   addition_to_counting   counting_to_addition 
#>              66.426447              63.701750              63.174626 
#> counting_with_straight   counting_to_straight      flags_with_visual 
#>              53.091740              50.382055              48.174095 
#>   straight_to_counting        flags_to_visual        visual_to_flags 
#>              47.246721              45.058315              37.515374 
#>       cubes_with_flags addition_with_straight   straight_with_visual 
#>              28.578723              28.512467              28.107703 
#>   addition_to_straight         cubes_to_flags         flags_to_cubes 
#>              27.340913              26.937281              26.904249 
#>   straight_to_addition     straight_to_visual    flags_with_straight 
#>              25.426929              24.902349              24.085941 
#>      flags_to_straight     visual_to_straight      straight_to_flags 
#>              22.649815              21.874736              21.467897 
#>    flags_with_sentence      flags_to_sentence      cubes_with_visual 
#>              17.720403              17.167791              16.415953 
#>        cubes_to_visual        visual_to_cubes   sentence_with_visual 
#>              15.439568              12.838972               9.620551 
#>     visual_to_sentence   counting_with_visual     counting_to_visual 
#>               7.927853               7.416599               7.057719 
#>    cubes_with_straight    counting_with_flags 
#>               6.818680               6.661301

Large indices often point to within-domain residual covariances (e.g. flags_with_visual) or cross-domain loadings that a single factor cannot represent. That pattern hints at multiple abilities rather than one g.

Three-factor alternative (no second MI)

Holzinger and Swineford’s data are usually modeled with three correlated abilities:

model3f = "
  spatial =~ visual + cubes + flags
  verbal  =~ paragrap + sentence + wordm
  speed   =~ addition + counting + straight
"
m3 = umxRAM(model3f, data = df, name = "Holzinger_3f")
umxSummary(m3, refModels = FALSE)
#> 
#> 
#> Table: Parameter loadings for model 'Holzinger_3f'
#> 
#> |   |name            |Estimate |SE   |CI                |type            |
#> |:--|:---------------|:--------|:----|:-----------------|:---------------|
#> |20 |p24_            |0.14     |0.04 |0.14 [0.06, 0.21] |Factor Cov      |
#> |21 |p23_            |0.21     |0.04 |0.21 [0.12, 0.29] |Factor Cov      |
#> |23 |p22_            |0.3      |0.06 |0.3 [0.19, 0.42]  |Factor Cov      |
#> |1  |p9_             |1.17     |0.21 |1.17 [0.76, 1.58] |Factor loading  |
#> |2  |p8_             |1.27     |0.16 |1.27 [0.95, 1.58] |Factor loading  |
#> |3  |p7_             |1        |0    |1 [1, 1]          |Factor loading  |
#> |4  |p6_             |0.98     |0.06 |0.98 [0.87, 1.1]  |Factor loading  |
#> |5  |p5_             |1        |0.06 |1 [0.89, 1.12]    |Factor loading  |
#> |6  |p4_             |1        |0    |1 [1, 1]          |Factor loading  |
#> |7  |p3_             |0.75     |0.12 |0.75 [0.52, 0.99] |Factor loading  |
#> |8  |p2_             |0.55     |0.11 |0.55 [0.34, 0.76] |Factor loading  |
#> |9  |p1_             |1        |0    |1 [1, 1]          |Factor loading  |
#> |19 |p21_            |0.32     |0.08 |0.32 [0.17, 0.48] |Factor Variance |
#> |22 |p20_            |0.72     |0.08 |0.72 [0.56, 0.89] |Factor Variance |
#> |24 |p19_            |0.59     |0.11 |0.59 [0.38, 0.81] |Factor Variance |
#> |25 |one_to_straight |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |26 |one_to_counting |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |27 |one_to_addition |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |28 |one_to_wordm    |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |29 |one_to_sentence |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |30 |one_to_paragrap |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |31 |one_to_flags    |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |32 |one_to_cubes    |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |33 |one_to_visual   |0        |0.06 |0 [-0.11, 0.11]   |Mean            |
#> |10 |p18_            |0.56     |0.09 |0.56 [0.38, 0.73] |Residual        |
#> |11 |p17_            |0.48     |0.09 |0.48 [0.3, 0.65]  |Residual        |
#> |12 |p16_            |0.67     |0.07 |0.67 [0.53, 0.82] |Residual        |
#> |13 |p15_            |0.3      |0.04 |0.3 [0.23, 0.37]  |Residual        |
#> |14 |p14_            |0.27     |0.03 |0.27 [0.2, 0.34]  |Residual        |
#> |15 |p13_            |0.27     |0.04 |0.27 [0.2, 0.34]  |Residual        |
#> |16 |p12_            |0.66     |0.07 |0.66 [0.51, 0.81] |Residual        |
#> |17 |p11_            |0.82     |0.08 |0.82 [0.67, 0.97] |Residual        |
#> |18 |p10_            |0.4      |0.09 |0.4 [0.23, 0.57]  |Residual        |
umxCompare(m3, m1)
#> 
#> 
#> Table: Table of Model Comparisons
#> 
#> |Model        | EP|Δ Fit   |Δ df |p       |      AIC|Δ AIC   |Compare with Model |Fit units |
#> |:------------|--:|:-------|:----|:-------|--------:|:-------|:------------------|:---------|
#> |Holzinger_3f | 30|        |     |        | 6905.374|0       |                   |-2lnL     |
#> |Holzinger_1f | 27|226.701 |3    |< 0.001 | 7126.075|220.701 |Holzinger_3f       |-2lnL     |
#> 
#> *Note*: EP = Estimated (i.e. free) parameters; Δ-2LL = change in -2 × Log-Likelihood of the model; Δ df = Change in degrees of freedom with respect to the comparison model; Δ AIC = Change in Akaike Information Criterion; 'Compared to' = The baseline model for this comparison.

The three-factor model fits substantially better. Remaining residual associations are typically smaller; run umxMI(m3) locally if you want that table.

Bifactor / multi-group (optional, local)

# Bifactor + umxMI, or multi-group one-factor by sex
modelBifactor = "
  g =~ visual + cubes + flags + paragrap + sentence + wordm + addition + counting + straight
  spatial =~ visual + cubes + flags
  verbal  =~ paragrap + sentence + wordm
  speed   =~ addition + counting + straight
  spatial ~~ 0*verbal + 0*speed + 0*g
  verbal  ~~ 0*speed + 0*g
  speed   ~~ 0*g
"
mBio = umxRAM(modelBifactor, data = df, name = "Holzinger_bifactor")
umxMI(mBio)

mMG = umxRAM("Holzinger_1f_MG", data = df, group = "Gender",
  umxPath("g", to = manifests),
  umxPath(v.m. = manifests),
  umxPath(v1m0 = "g")
)
umxMI(mMG)

Summary

Model Role
One-factor Baseline misspecification; one umxMI() call
Three-factor Theory-driven improvement via umxCompare()
Bifactor / multi-group Optional local demos

Take-home: Use umxMI() to diagnose where a model misfits; prefer umxCompare() on theory-driven competing models for inference.

References

Holzinger, K. J., & Swineford, F. (1939). A study in factor analysis: The stability of a bi-factor solution. Supplementary Educational Monographs, 48. University of Chicago Press.