Applications of Item Response Theory to Clinical ADHD Research: Analysis of the ADHDRS-IV and Modeling Response to Treatments and Item-Level Treatment Sensitivity

In order to evaluate an IRT model for the ADHDRS-IV, first descriptive statistics will be calculated. Computed descriptive statistics will include mean, standard deviation, skewness, and kurtosis, and response category frequencies. Response category frequencies, or how many individuals responded within each of the 4^9 possible response patterns are important for determining how well the sample is covering the pattern of possible responses. Patterns and frequency of missing data will be examined to assess possible non-random missingness.

For completeness, classical test theory statistics will be estimated. Cronbach’s coefficient alpha (Cronbach, 1951) will be used to examine internal consistency. Minimum acceptable values are 0.70 to 0.80 for measurement at the group level and 0.90 to 0.95 at the individual level. Inter-item correlations and item-scale correlations will also be calculated.

Prior to fitting the IRT model, it will be important to evaluate the three assumptions of the model, namely unidimensionality, local independence, and monotonicity. Prior research has established the multidimensionality of the items that constitute the inattention and hyperactivity/impulsivity diagnoses (Glutting, Youngstrom, & Watkins, 2005; American Psychiatric Association, 2013; DuPaul, Anastopoulos, Power, Reid, Ikeda, & McGoey, 1998; Hudziak et al., 1998; Martel, Schimmack, Nikolas, & Nigg, 2015). However, it is important to confirm the appropriate dimensionality of the ADHD construct. Because significant substantive and theoretical literature has established the bi-dimensional structure of item factor models for ADHD, three confirmatory IRT models will be fit. There is no expectation that both hyperactivity/impulsivity and inattention items are best represented by a single factor, but a unidimensional IRT model will be fit for completeness. A bifactor model (Gibbons & Hedeker, 1992) will also be fit to explore the possibility of two specific factors, inattention and hyperactivity/impulsivity, and one general dimension that represents general ADHD trait variation. Loadings on the specific factors and general dimension will be examined to determine if the bifactor structure is warranted. If it appears that a general dimension does not help to explain the trait of ADHD, a correlated traits IRT model will be fit to further evaluate the possible presence of a general dimension. If the correlation between inattention and hyperactivity/impulsivity is sufficiently low (e. g. r = .1 to .4), it is more parsimonious to simply fit two separate unidimensional IRT models (Reise, Morizot, & Hays, 2007). To assess for local dependence, two unidimensional IRT models will be fit to the data, one for the inattention items and one for hyperactivity/impulsivity items and standardized Chen & Thissen (1997) local dependence chi-square indices will be examined. High residual associations between items may indicate un-modeled latent dimensions. Finally, monotonicity can be assessed by evaluating rest score plots (Junker & Sitjsma, 2000).

Item response theory will be used to model the psychometric properties of the ADHDRS-IV including item discrimination, thresholds, and the unique information about the latent trait of ADHD provided by items. Specifically, a graded response, unidimensional model for symptoms of inattention and hyperactivity/impulsivity will be modeled separately as supported by literature. The requested studies are all multi-site trials and data was collected across the U.S. A multilevel item factor model will allow the investigator to model the nested nature of the data, where study participants (level-1 units) were nested within sites (level-2 units). The two-level data will then be specifically structured using between- and within-site latent dimensions.

Models will likely be estimated using maximum marginal likelihood via the Bock-Aitkin EM algorithm (Bock & Aitkin, 1981), which will also handle missing data. Several indices of model fit will be reported and used to assess the fit of the model. Due the possibility of many response patterns (4^9) and a sparse contingency table, the M2 statistic will be reported (Cai & Hansen, 2013). Root mean square error of approximation (RMSEA; Steiger & Lind, 1980) will also be reported in addition to marginal fit (chi-square) and LD chi-square statistics for further evaluation and discussion of model fit.

Sub-group differences across gender and age will be explored using differential item functioning using a multi-step Wald chi-square procedure (Woods, Cai, & Wang, 2013; Langer, 2008; Cai et al., 2011)). In addition, DIF may also be computed within and between sites for the ADHDRS-IV items. This would indicate possible differences in probability of response or location for an item given the same latent traits score depending on the site of participation. Multiple DIF comparisons inflates the rate of type I error. This scenario would lead us to believe that invariance exists, when in fact it does not. To control the error rate, the false discovery rate procedure (Benjamini & Hochberg, 1995) will be used.

Study 2
The central aim of study 2 is to estimate the effect of several different treatment approaches on specific ADHD symptoms. A multiple group multilevel two-tier (MTT) item factor model will be used to estimate latent change scores in the ADHD construct over treatment (Cai, 2010). This is minimally a bivariate regression model where the outcome variables are latent and observed items are categorical and multivariate. The MTT model will also be used to assess differences in item-level treatment sensitivity, or how items may differentially respond to specific treatments. Treatment assignment will be entered as a covariate in the model.

There are no definitive rules regarding sample size in IRT. That being said, there are several thoughts on sample size to consider when modeling. More complex models with more parameters need larger samples. In addition, when the samples more closely meet the assumptions of IRT models (normality, monotonicity, unidimensionality), parameter estimates will be more accurate (Edelen & Reeve, 2007). Finally, as sample size increases, standard errors generally decrease, giving us greater confidence in the precision of our estimates. In order to estimate a more complex model with more parameters, including the two-tier IRT model (Cai, 2010), for treatment studies with smaller sample sizes, item parameter estimates can be generated from a large pre-treatment sample and then used in subsequent analyses so that fewer parameters need be estimated. Study 1 serves this purpose.

The sample sizes of each treatment study will not be sufficiently large to both estimate parameters and change in the latent trait over time. Item discrimination and threshold parameters from study 1 will be used to reduce the number of parameters estimated using the smaller treatment samples and also serve to fix the scale of the primary dimensions so that both pretreatment and post-treatment means and variances for the primary latent dimensions can be estimated.

The multiple group MTT item factor model will contain several possible sources of differential item functioning. The first is within treatment DIF, or DIF between parameters of an item measured at pretreatment and post-treatment. The second is between-treatment DIF, or how inattention item parameters at post-treatment may differ between groups. Due to the expected change in ADHD inattention symptoms that may accompany treatment, it is an appropriate next step to look more closely at how inattention item parameters might change over time. DIF can reveal how probability of response to an item or location of the item may differ between treatment groups. Another possible layer of DIF may exist due to the multilevel nature of the data, namely post-treatment DIF within sites and between sites.

To test for within- and between-treatment DIF, many comparisons will need to be made. As in study 1, a series of Wald tests can be used to examine DIF. Even if item-level DIF is identified, the overall impact on total scores may not be large. The practical significance of DIF will need to be judged within the treatment framework. Again, error rate will be controlled using the false discovery rate procedure (Benjamini & Hochberg, 1995).

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300.

Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46 , 443–459.

Cai, L. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581-612.

Cai, L., & Hansen, M. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66(2), 245-276.

Cai, L., Thissen, D., & du Toit, S. H. C. (2011). IRTPRO: Flexible, multidimensional, multiple
categorical IRT modeling [Computer software]. Lincolnwood, IL: Scientific Software
International.

Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics,22(3), 265-289.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.

DuPaul, G. J., Anastopoulos, A. D., Power, T. J., Reid, R., Ikeda, M. J., & McGoey, K. E. (1998). Parent ratings of Attention-Deficit/Hyperactivity Disorder symptoms: Factor structure and normative data. Journal of Psychopathology and Behavioral Assessment, 20(1), 83-102.

Edelen, M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16(1), 5-18.

Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57(3), 423-436.

Glutting, J. J., Youngstrom, E. A., & Watkins, M. W. (2005). ADHD and college students: exploratory and confirmatory factor structures with student and parent data. Psychological Assessment, 17(1), 44-55.

Hudziak, J. J., Heath, A. C., Madden, P. F., Reich, W., Bucholz, K. K., Slutske, W., Bierut, L. J., Neuman, R. J., & Todd, R. D. (1998). Latent class and factor analysis of DSM-IV ADHD: a twin study of female adolescents. Journal of the American Academy of Child & Adolescent Psychiatry, 37(8), 848-857.

Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.Applied Psychological Measurement, 25(3), 258-272.

Langer, M. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation (Unpublished doctoral dissertation).

Martel, M. M., Schimmack, U., Nikolas, M., & Nigg, J. T. (2015). Integration of symptom ratings from multiple informants in ADHD diagnosis: a psychometric model with clinical utility. Psychological Assessment. Advance online publication.

Reise, S. P., Morizot, J., & Hays, R. D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16(1), 19-31.

Steiger, J. H. & Lind, J. C. (1980). Statistically Based Tests for the Number of Common Factors. Paper presented at the annual meeting of the Psychometric Society, May, Iowa City, IA.

Woods, C. M., Cai, L., & Wang, M. (2013). The Langer-Improved Wald test for DIF testing with multiple groups: evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532-547.