ORIGINAL ARTICLE
A literature review of model fit and model comparisons with confirmatory factor analysis: Formalizing the informal in organizational science
Matt C. Howard,
Melanie Boudreaux,
Joshua Cogswell,
Kelly G. Manix,
Matthew T. Oglesby,
Corresponding Author
Matt C. Howard
Mitchell College of Business, University of South Alabama, Mobile, Alabama, USA
Correspondence
Matt C. Howard, Mitchell College of Business, University of South Alabama, 5811 USA Drive S., Rm. 337, Mobile, AL 36688, USA.
Email: [email protected]
Search for more papers by this authorMelanie Boudreaux
Al Danos College of Business Administration, Nicholls State University, Thibodaux, Louisiana, USA
Search for more papers by this authorJoshua Cogswell
Al Danos College of Business Administration, Nicholls State University, Thibodaux, Louisiana, USA
Search for more papers by this authorKelly G. Manix
Jennings A. Jones College of Business, Middle Tennessee State University, Murfreesboro, Tennessee, USA
Search for more papers by this authorMatthew T. Oglesby
Sanders College of Business and Technology, University of North Alabama, Florence, Alabama, USA
Search for more papers by this authorMatt C. Howard,
Melanie Boudreaux,
Joshua Cogswell,
Kelly G. Manix,
Matthew T. Oglesby,
Corresponding Author
Matt C. Howard
Mitchell College of Business, University of South Alabama, Mobile, Alabama, USA
Correspondence
Matt C. Howard, Mitchell College of Business, University of South Alabama, 5811 USA Drive S., Rm. 337, Mobile, AL 36688, USA.
Email: [email protected]
Search for more papers by this authorMelanie Boudreaux
Al Danos College of Business Administration, Nicholls State University, Thibodaux, Louisiana, USA
Search for more papers by this authorJoshua Cogswell
Al Danos College of Business Administration, Nicholls State University, Thibodaux, Louisiana, USA
Search for more papers by this authorKelly G. Manix
Jennings A. Jones College of Business, Middle Tennessee State University, Murfreesboro, Tennessee, USA
Search for more papers by this authorMatthew T. Oglesby
Sanders College of Business and Technology, University of North Alabama, Florence, Alabama, USA
Search for more papers by this authorFirst published: 03 December 2024
Joshua Cogswell, Kelly Manix, and Matthew Oglesby contributed equally to the manuscript, and their names appear in alphabetical order.
We would like to thank Jeffrey Lovelace, Andrew Hanna, Janaki Gooty, and especially Brett Neely for their feedback on an earlier version of the current article.
Funding Information: No funding was received in association with the current wor
Abstract
Researchers often stray from recommendations provided by simulation studies when conducting confirmatory factor analysis (CFA), causing unwieldy applications of the analysis and diminished confidence in published results. We introduce three particularly important informal practices associated with (1) alternative interpretations of model fit, (2) the use of inadvisable combinations of fit indices, and (3) the failure to conduct effective model comparisons. We then review over 2000 CFAs in premier organizational science journals. Our results support that researchers widely engage in all three informal practices. To address this tension, we (1) formalize modern interpretations of model fit by providing percentile ranges of indices in published articles, such that researchers can make relative and continuous assessments of model fit. We (2) emphasize the importance of assessing multiple recommended fit indices together to provide complete depictions of model soundness. Lastly, we (3) demonstrate the necessity to perform appropriate model comparisons, including the assessment of more complex models.
CONFLICT OF INTEREST STATEMENT
The authors have no conflicts of interest to disclose.
Open Research
DATA AVAILABILITY STATEMENT
All data associated with the current article are provided in the supplemental materials.
Supporting Information
| Filename | Description |
|---|---|
| apps12592-sup-0001-Supplemental_Material_A.docxWord 2007 document , 15.5 KB | Data S1. Supporting Information |
| apps12592-sup-0002-Supplemental_Material_B.docxWord 2007 document , 41.3 KB | Data S2. Supporting Information |
| apps12592-sup-0003-Supplemental_Material_C.docxWord 2007 document , 35.6 KB | Data S3. Supporting Information |
| apps12592-sup-0004-Supplemental_Material_D.docxWord 2007 document , 24.2 KB | Data S4. Supporting Information |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- Aguinis, H., Gottfredson, R. K., & Joo, H. (2013). Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods, 16(2), 270–301. https://doi.org/10.1177/1094428112470848
- Ainur, A. K., Sayang, M. D., Jannoo, Z., & Yap, B. W. (2017). Sample size and non-normality effects on goodness of fit measures in structural equation models. Pertanika Journal of Science & Technology, 25(2), 575–586.
- Banks, G. C., Field, J. G., Oswald, F. L., O'Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2019). Answers to 18 questions about open science practices. Journal of Business and Psychology, 34, 257–270. https://doi.org/10.1007/s10869-018-9547-8
- Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815–824. https://doi.org/10.1016/j.paid.2006.09.018
- Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12(1), 41–75. https://doi.org/10.1207/s15328007sem1201_3
- Bollen, K. A. (1990). Overall fit in covariance structure models: Two types of sample size effects. Psychological Bulletin, 107(2), 256–259. https://doi.org/10.1037/0033-2909.107.2.256
- Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431–449. https://doi.org/10.1037/a0038047
- Brown, T. A. (2015). Confirmatory factor analysis for applied research. Guilford publications.
- Brown, T. A., & Moore, M. T. (2012). Confirmatory factor analysis. In Handbook of structural equation modeling ( 2nd ed., Vol. 361). (p. 379). Guilford Press.
- Browne, M. W., MacCallum, R. C., Kim, C. T., Andersen, B. L., & Glaser, R. (2002). When fit indices and residuals are incompatible. Psychological Methods, 7(4), 403–421. https://doi.org/10.1037/1082-989X.7.4.403
- Calder, B. J., Brendl, C. M., Tybout, A. M., & Sternthal, B. (2021). Distinguishing constructs from variables in designing research. Journal of Consumer Psychology, 31(1), 188–208. https://doi.org/10.1002/jcpy.1204
- Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494. https://doi.org/10.1177/0049124108314720
- Cortina, J. M., Green, J. P., Keeler, K. R., & Vandenberg, R. J. (2017). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20(3), 350–378. https://doi.org/10.1177/1094428116676345
- Credé, M., & Harms, P. (2019). Questionable research practices when using confirmatory factor analysis. Journal of Managerial Psychology., 34, 18–30. https://doi.org/10.1108/JMP-06-2018-0272
- Credé, M., & Harms, P. D. (2015). 25 years of higher-order confirmatory factor analysis in the organizational sciences: A critical review and development of reporting recommendations. Journal of Organizational Behavior, 36(6), 845–872. https://doi.org/10.1002/job.2008
- Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. https://doi.org/10.1037/1082-989X.1.1.16
- Ding, L., Velicer, W. F., & Harlow, L. L. (1995). Effects of estimation methods, number of indicators per factor, and improper solutions on structural equation modeling fit indices. Structural Equation Modeling: A Multidisciplinary Journal, 2(2), 119–143. https://doi.org/10.1080/10705519509540000
- Douma, J. C., & Shipley, B. (2023). Testing model fit in path models with dependent errors given non-normality, non-linearity and hierarchical data. Structural Equation Modeling: a Multidisciplinary Journal, 30(2), 222–233. https://doi.org/10.1080/10705511.2022.2112199
- Fan, X., & Sivo, S. A. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42(3), 509–529. https://doi.org/10.1080/00273170701382864
- Goretzko, D., Siemund, K., & Sterner, P. (2024). Evaluating model fit of measurement models in confirmatory factor analysis. Educational and Psychological Measurement., 84, 123–144. https://doi.org/10.1177/00131644231163813
- Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis. Pearson.
- Hancock, G. R., & Mueller, R. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educational and Psychological Measurement, 71(2), 306–324. https://doi.org/10.1177/0013164410384856
- Harrington, D. (2009). Confirmatory factor analysis. Oxford University Press.
- Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16(3), 319–336. https://doi.org/10.1037/a0024917
- Heene, M., Hilbert, S., Freudenthaler, H. H., & Bühner, M. (2012). Sensitivity of SEM fit indices with respect to violations of uncorrelated errors. Structural Equation Modeling, 19(1), 36–50. https://doi.org/10.1080/10705511.2012.634710
- Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967–988. https://doi.org/10.1177/014920639502100509
- Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121. https://doi.org/10.1177/109442819800100106
- Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modeling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53–60.
- Howard, M. C. (2023). A systematic literature review of exploratory factor analyses in management. Journal of Business Research, 164, 113969. https://doi.org/10.1016/j.jbusres.2023.113969
- Howard, M. C., & Henderson, J. (2023). A review of exploratory factor analysis in tourism and hospitality research: Identifying current practices and avenues for improvement. Journal of Business Research, 154, 113328. https://doi.org/10.1016/j.jbusres.2022.113328
- Howard, M. C., Boudreaux, M., & Oglesby, M. (2024). Can Harman's single-factor test reliably distinguish between research designs? Not in published management studies. European Journal of Work and Organizational Psychology, 33(6), 790–804. https://doi.org/10.1080/1359432X.2024.2393462
10.1080/1359432X.2024.2393462 Google Scholar
- Hu, L. T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989X.3.4.424
- Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
- Hurley, A. E., Scandura, T. A., Schriesheim, C. A., Brannick, M. T., Seers, A., Vandenberg, R. J., & Williams, L. J. (1997). Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives. Journal of Organizational Behavior, 18, 667–683. https://doi.org/10.1002/(SICI)1099-1379(199711)18:6<667::AID-JOB874>3.0.CO;2-T
- Iacobucci, D. (2010). Structural equations modeling: Fit indices, sample size, and advanced topics. Journal of Consumer Psychology, 20(1), 90–98. https://doi.org/10.1016/j.jcps.2009.09.003
- Jackson, D. L., Gillaspy, J. A. Jr., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6–23. https://doi.org/10.1037/a0014694
- Jobst, L. J., Auerswald, M., & Moshagen, M. (2022). The effect of latent and error non-normality on measures of fit in structural equation modeling. Educational and Psychological Measurement, 82(5), 911–937. https://doi.org/10.1177/00131644211046201
- Kenny, D. (2023). Measuring model fit. DavidAKenny.net. Retrieved from: https://davidakenny.net/cm/fit.htm
- Kenny, D. A., Kaniskan, B., & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44(3), 486–507.
- Kenny, D. A., & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10(3), 333–351. https://doi.org/10.1207/S15328007SEM1003_1
- Kline, R. B. (2023). Principles and practice of structural equation modeling. Guilford Publications.
- Koran, J. (2020). Indicators per factor in confirmatory factor analysis: More is not always better. Structural Equation Modeling: a Multidisciplinary Journal, 27(5), 765–772. https://doi.org/10.1080/10705511.2019.1706527
- Lance, C. E., & Vandenberg, R. J. (2002). Confirmatory factor analysis. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 221–254). Jossey-Bass/Wiley.
- MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130
- Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103(3), 391–410. https://doi.org/10.1037/0033-2909.103.3.391
- Marsh, H. W., Hau, K. T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33(2), 181–220. https://doi.org/10.1207/s15327906mbr3302_1
- Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2
- McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43–52. https://doi.org/10.1080/00223891.2017.1281286
- McNeish, D., & Wolf, M. G. (2021). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61.
- McNeish, D., & Wolf, M. G. (2023). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61–88. https://doi.org/10.1037/met0000425
- Miles, J., & Shevlin, M. (2007). A time and a place for incremental fit indices. Personality and Individual Differences, 42(5), 869–874. https://doi.org/10.1016/j.paid.2006.09.022
- Nye, C. D. (2022). Reviewer resources: Confirmatory factor analysis. Organizational Research Methods, 26(4), 608–628.
- Perry, J. L., Nicholls, A. R., Clough, P. J., & Crust, L. (2015). Assessing model fit: Caveats and recommendations for confirmatory factor analysis and exploratory structural equation modeling. Measurement in Physical Education and Exercise Science, 19(1), 12–21. https://doi.org/10.1080/1091367X.2014.952370
- Peugh, J., & Feldon, D. F. (2020). “How well does your structural equation model fit your data?”: Is Marcoulides and Yuan's equivalence test the answer? CBE—Life Sciences Education, 19(3), es5. https://doi.org/10.1187/cbe.20-01-0016
- Podsakoff, P. M., Podsakoff, N. P., Williams, L. J., Huang, C., & Yang, J. (2024). Common method bias: It's bad, it's complex, it's widespread, and it's not easy to fix. Annual Review of Organizational Psychology and Organizational Behavior, 11, 17–61. https://doi.org/10.1146/annurev-orgpsych-110721-040030
- Preacher, K. J. (2003). The role of model complexity in the evaluation of structural equation models. The Ohio State University.
- Rigdon, E. E. (1994). Calculating degrees of freedom for a structural equation model. Structural Equation Modeling: A Multidisciplinary Journal, 1(3), 274–278. https://doi.org/10.1080/10705519409539979
- Saris, W. E., Satorra, A., & van der Veld, W. M. (2009). Testing structural equation models or detection of misspecifications. Structural Equation Modeling, 16(4), 561–582. https://doi.org/10.1080/10705510903203433
- Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932. https://doi.org/10.1177/0013164412452564
- Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74.
- Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58(7), 935–943. https://doi.org/10.1016/j.jbusres.2003.10.007
- Shevlin, M., & Miles, J. N. (1998). Effects of sample size, model specification and factor loadings on the GFI in confirmatory factor analysis. Personality and Individual Differences, 25(1), 85–90. https://doi.org/10.1016/S0191-8869(98)00055-5
- Shi, D., DiStefano, C., Maydeu-Olivares, A., & Lee, T. (2022). Evaluating SEM model fit with small degrees of freedom. Multivariate Behavioral Research, 57(2–3), 179–207. https://doi.org/10.1080/00273171.2020.1868965
- Shi, D., Maydeu-Olivares, A., & DiStefano, C. (2018). The relationship between the standardized root mean square residual and model misspecification in factor analysis models. Multivariate Behavioral Research, 53(5), 676–694. https://doi.org/10.1080/00273171.2018.1476221
- Somaraju, A. V., Nye, C. D., & Olenick, J. (2022). A review of measurement equivalence in organizational research: What's old, what's new, what's next? Organizational Research Methods, 25(4), 741–785. https://doi.org/10.1177/10944281211056524
- Stamenkov, G. (2023). Recommendations for improving research quality: Relationships among constructs, verbs in hypotheses, theoretical perspectives, and triangulation. Quality & Quantity, 57(3), 2923–2946. https://doi.org/10.1007/s11135-022-01461-2
10.1007/s11135-022-01461-2 Google Scholar
- Tomarken, A. J., & Waller, N. G. (2003). Potential problems with “well fitting” models. Journal of Abnormal Psychology, 112(4), 578–598. https://doi.org/10.1037/0021-843X.112.4.578
- Tomarken, A. J., & Waller, N. G. (2005). Structural equation modeling: Strengths, limitations, and misconceptions. Annual Review of Clinical Psychology, 1, 31–65. https://doi.org/10.1146/annurev.clinpsy.1.102803.144239
- Vicente-Saez, R., & Martinez-Fuentes, C. (2018). Open Science now: A systematic literature review for an integrated definition. Journal of Business Research, 88, 428–436. https://doi.org/10.1016/j.jbusres.2017.12.043
- West, S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In Handbook of structural equation modeling (Vol. 1(1) (pp. 209–231). Guilford Press.
- Whittaker, T. A. (2012). Using the modification index and standardized expected parameter change for model modification. The Journal of Experimental Education, 80(1), 26–44. https://doi.org/10.1080/00220973.2010.531299
- Widaman, K. F., & Thompson, J. S. (2003). On specifying the null model for incremental fit indices in structural equation modeling. Psychological Methods, 8(1), 16–37. https://doi.org/10.1037/1082-989X.8.1.16
- Williams, L. J., Ford, L. R., & Nguyen, N. (2004). Basic and advanced measurement models for confirmatory factor analysis. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 366–389). Blackwell Publishing.
10.1002/9780470756669.ch18 Google Scholar
- Wolf, M. G., & McNeish, D. (2023). dynamic: An R package for deriving dynamic fit index cutoffs for factor analysis. Multivariate Behavioral Research, 58, 189–194. https://doi.org/10.1080/00273171.2022.2163476
- Ximénez, C., Revuelta, J., & Castañeda, R. (2022). What are the consequences of ignoring cross-loadings in bifactor models? A simulation study assessing parameter recovery and sensitivity of goodness-of-fit indices. Frontiers in Psychology, 13, 923877. https://doi.org/10.3389/fpsyg.2022.923877
- Yin, Y., Shi, D., & Fairchild, A. J. (2023). The effect of model size on the root mean square error of approximation (RMSEA): The nonnormal case. Structural Equation Modeling: A Multidisciplinary Journal, 30(3), 378–392. https://doi.org/10.1080/10705511.2022.2127729
- Yuan, K. H., Chan, W., Marcoulides, G. A., & Bentler, P. M. (2016). Assessing structural equation models by equivalence testing with adjusted fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 23(3), 319–330. https://doi.org/10.1080/10705511.2015.1065414
