Elsevier

Journal of Affective Disorders

Volume 227, February 2018, Pages 698-704
Journal of Affective Disorders

Research paper
Distinguishing transient versus stable aspects of depression in New Zealand Pacific Island children using Generalizability Theory

https://doi.org/10.1016/j.jad.2017.11.075Get rights and content

Highlights

  • G Theory provides a method for estimating state and trait components of measures.

  • We used G on the brief Children's Depression Inventory with Pacific Island children.

  • The CDI-10 had acceptable generalizability across 3 occasions.

  • About 1/3 of variance was due to dynamic and 2/3 to enduring aspects of depression.

Abstract

Background

The distinction between temporary versus enduring or state/trait aspects of depression is important. More precise distinction would improve understanding of the aetiology of depression and those aspects most amenable to intervention thus identifying more homogeneous, dynamic targets for clinical trials. Generalizability Theory has been proposed as useful for disentangling state and trait components of psychopathology.

Methods

We applied Generalizability Theory to determine the relative contributions of temporary and enduring aspects of depression in a widely used screening measure of depression the – 10-item Children's Depression Inventory (CDI-10; Kovacs, 1985). Participants were children of Pacific Island descent living in New Zealand (n = 668). Data were collected at ages – 9, 11, and 14 years.

Results

The CDI-10 demonstrated acceptable generalizability across occasions (G = 0.79) with about one third of variance in total scores attributed to temporary and two thirds to more enduring aspects of depression. There were no other significant sources of error variance. Two items were identified as more sensitive than the remaining eight to more dynamic symptoms.

Limitations

Studies with briefer test-retest intervals are warranted. Use of this Pacific Island cohort limits generalizability of findings to other cultures and ethnicities. No data were collected on whether participants had received intervention for depression.

Conclusions

While the CDI-10 reliably measures both stable and transient aspects of depression in children, the scale does not permit clear distinction between them. We advocate application of Generalizability Theory for developing state/trait depression measures and determining which existing measures are most suitable for capturing modifiable features of depression.

Introduction

The distinction between dynamic and more stable aspects of behavior is well established in psychology (Chaplin et al., 1988). Generally, relatively stable or enduring aspects of an individual's behavior are referred to as a trait, while more dynamic aspects are referred to as a state, meaning changeable behavior that is much more context bound or situation specific. The distinction between such stable and dynamic aspects of behavior has sometimes been demonstrated by comparing test - retest correlations for the state and trait subscales of a measure, e.g., the State-Trait Anxiety Inventory (STAI; Gaudry et al., 1975), with a higher correlation expected for trait than state components. For example, Barnes et al. (2002) reviewed the reliability of the STAI and found a mean test-retest correlation of 0.88 (range 0.82–0.94) for trait anxiety and 0.70 (range 0.34–0.96) for state anxiety, based on seven studies reporting test-retest reliability. An obvious limitation of this approach to distinguishing state from trait is that it uses total score correlations and fails to examine the relative strengths of individual items.

An alternative approach to assessing state versus trait has been to use structural equation modelling to compare the covariance structures of a state/trait measure on at least two separate occasions (Cole et al., 2005). Using this “latent trait-state-occasion model” with the Beck Depression Inventory II (BDI-II) in two large samples of young people Wu reported a trait component accounting for more than 50% of variance and an ‘occasion-specific factor’ that explained between 7% and 12% of variance (Wu, 2016, p.39). LaGrange et al. (2011) applied the latent trait-state-occasion model to four measures of depressive cognitions and symptoms in a longitudinal study of 515 children and adolescents and found three of these measures characterized by “two types of longitudinal factors: a time invariant (or trait-like) factor and a series of time-varying (more state-like) factors” (p.13). Of particular relevance to the present study was their fourth measure - the Children's Depression Inventory - was almost entirely accounted for by time varying or state like variance. While this approach has the advantage that it can compare the strength of the loadings of individual items on state or trait factors at different points in time it fails to partial out the precise proportion of variance across occasions due to the person, the occasion and the item (as well as their interactions). A method which does exactly this, namely to account for all major sources of variance in a measurement situation, and arguably (Medvedev et al., 2017) should be the preferred technique for demonstrating a state-trait distinction is Generalizability Theory.

Generalizability or G Theory was developed by Cronbach and represents a major advance upon Classical Test Theory (CTT) particularly in regard to evaluating reliability (Cronbach et al., 1963). While CTT conceptualizes test scores as the sum of a true score and error variance, G Theory uses analysis of variance (ANOVA) to calculate precise estimates for the error variance due to each important measurement facet, where facet refers to a distinct element that might influence variance and error in test scores in any testing situation. For example, the persons tested, the test items and the testing occasion are three examples of facets. Hence, whereas CTT restricts analysis of reliability and error variance to a single element such as the test items (i.e., Cronbach's alpha), the occasion (test-retest) or the rater (inter-rater), G Theory permits the researcher to break reliability down into all the important facets contributing to measurement error in a single analysis. In G terms the variance associated with participants or persons is considered the central concern and is known as the differentiation facet with other facets (e.g., items, occasion, rater) viewed as sources of measurement error.

An important step in a Generalizability analysis is the calculation of individual variance components for each facet (e.g., person, item, occasion) in the measurement situation. These variance components are first calculated by a conventional ANOVA and corrected by a method known as ‘Whimbey's correction' which accounts for the type of sampling involved (i.e. random, fixed or random finite). This also includes calculation of a G coefficient which represents how generalizable the test scores are across different situations. Bloch and Norman (2012, p. 968) describe the G coefficient as the ratio of signal to noise or ‘true’ variance to ‘true + error variance’ and provide details on its derivation and calculation. In most Generalizability studies there are two stages known as the G-study and the D-study. The G-study involves a standard factorial ANOVA with the calculation of variance components for each facet and for their interactions (rather than the significance tests typically associated with an ANOVA). The D-study allows the researcher to then estimate the impact on reliability of variations in different facets such as increasing the number of participants or the number of items in a scale.

As an illustration, medical education is one setting where G theory has been widely used (e.g., Prion et al., 2016). For example, when nursing students are examined on clinical skills, they might encounter ten structured clinical problems (or stations) that they are required to respond to in the presence of an examiner who rates their performance on a 1–10 scale for each problem. Assuming a different examiner (i.e. rater) at each station, a G Theory approach permits us to calculate the variance component for each facet – namely student, station, and rater - and their interactions. Consider another example from a mental health setting. In examining the inter-rater reliability of a new rating scale for assessing the severity of depression in outpatients we might have the following facets: Person (the differentiation facet), rater, and occasion. The calculation of variance components for each facet and their interactions allows us to determine precisely how much variance is due to the patient (Person facet) and changes in their condition (Person × Occasion) as well as the reliability of raters. High inter-rater reliability will be reflected in a very small variance component for the facet Rater. In the present article we argue that G Theory may have particular value for distinguishing between state and trait or stable versus enduring aspects of behavior and depression in particular.

A key issue for measuring change in symptoms of psychopathology is the extent to which the underlying construct is stable or changes over time – the state-trait distinction. This distinction between stable and dynamic symptoms has been applied less often for conceptualizing depression than for anxiety. However, there has been increasing interest recently in separating dynamic versus more stable aspects of depression from both neurobiological and personality perspectives (Bhagwagar et al., 2002, Graham et al., 2013, Natoli et al., 2016, Wu, 2016). In a recent psychometric study, Wu (2016) tested a latent state-trait-occasion model using the Beck Depression Inventory II (BDI-II) in two cohorts, of adolescents and young adults, concluding that the BDI-II measures “both trait-like and occasion-specific variances for individuals during late adolescence to early adulthood” (p.48). Lakes and Hoyt (2009) have argued that G Theory is a particularly useful conceptual framework for measuring the stable and transient components of a latent construct among adolescents where development is paramount. A recent paper by Chavez et al. (2016) demonstrated this approach with the Adolescent Quality of Life-Mental Health Scale (AQOL-MHS) in adolescents attending mental health clinics assessed on three occasions over eight months. The focus of that study was on the reliability of change scores and the authors used a method developed by Cranford et al. (2006) based on G Theory to compare conventional estimates of reliability (Cronbach's alpha, test-retest) with GT estimates. The authors observed that the reliability of change assessed by G Theory for the three subscales of the AQOL-MHS was adequate but consistently lower than Cronbach's alpha for each time point. They concluded Cronbach's alpha is sufficient when comparing scores across people at the same point in time but the reliability of change scores must be considered when comparing the change in persons across multiple time points.

The Children's Depression Inventory is a widely used measure of depression in children and adolescents. It is a 27-item self-report questionnaire for quantifying the severity of symptoms of depression in young people aged 7–17 years. We are aware of only one previous article using G Theory to investigate the psychometric characteristics of the CDI (Crowley et al., 1994). In that study they administered the full 27-item CDI twice, with a 28-week retest interval, to 164 children aged 11–16 years (mean age 12.6) in Texas schools. Crowley et al. (1994) found a relatively low generalizability coefficient for a single CDI administration (0.63) compared with repeat testing (0.81) and advocated multiple assessments before diagnosing depression in a child. They noted that the largest source of error in their analysis was the Persons × Items interaction suggesting that the CDI items were interpreted differently by different participants. The variance component for the P × I interaction (17.3%) in that analysis was notably larger than that for Persons (13.4%), Items (5.8%) and Persons × Time (5.2%).

In a recent paper Medvedev et al. (2017) argued that G Theory offers unique advantages for separating the dynamic from stable dimensions of a latent trait. They argued that the variance due to the person facet in a repeated measures test situation represents trait behavior and that state behavior is indexed by the variance due to the person × occasion interaction. Moreover, a state measure should be characterized by low person and low occasion variance, but a relatively large person × occasion interaction variance component. Conversely, a trait measure should display large amounts of variance due to the person facet and low person × occasion variance. In the same article Medvedev et al. proposed a new index for characterizing the state (state component index, SCI) and trait (trait component index, TCI) components of a measure. G Theory produced variance components consistent with this interpretation and thereby demonstrated the construct validity of trait and state behavior. In the present paper, we use G Theory to examine the short version of the CDI for state versus trait characteristics in a large cohort of Pacific children born in Auckland, New Zealand. The brief CDI was originally chosen for this longitudinal cohort study because its brevity made it practical to include in a much longer battery of assessment tools. The availability of data from three points in time made it particularly suitable for the present Generalizability analysis. The aims of the present research were: (1) to examine the psychometric properties of the 10-item version of the CDI using G Theory and (2) to explore the degree to which state and trait or stable versus transient aspects of depression can be quantified for this scale.

Section snippets

Participants

All participants were children of Pacific Island descent living in New Zealand participating in the longitudinal Pacific Islands Families Study https://niphmhr.aut.ac.nz/research-centres/centre-for-pacific-health-and-development-research/pacific-islands-families-study. The Pacific Islands Families (PIF) Study is a longitudinal investigation of Pacific children born in Auckland, New Zealand, and their parents. All potential participants from Middlemore hospital in South Auckland were selected

Results

We report the results of the analyses for the full sample of 668 children first and then report the sub-group analysis for the 244 children with a score ≥ 4 on the brief CDI. Descriptive statistics for the CDI data collected at three time points with two and three year intervals are presented in Table 2. The average internal consistency or Cronbach's alpha over three occasions was 0.71 and test-retest reliability with a two and five year interval was 0.41 and 0.22 respectively (both p< 0.001).

Discussion

This study examined the psychometric properties of the 10-item version of the CDI-10 using G Theory and to determine the extent to which the scale has the capacity to parse depression as a temporary or enduring construct. The CDI-10 was initially developed as a rapid screening tool to quantify the severity of depressive symptoms in children for clinical and research purposes (Kovacs, 1985). The measure was found to have a G coefficient of 0.79, which is very close to the accepted criterion of

Acknowledgements

This research was conducted within the School of Public Health and Psychosocial Studies of the Faculty of Health and Environmental Sciences at the Auckland University of Technology. This article resulted from a meeting of the authors in April 2017 funded by a Royal Society of New Zealand Catalyst (Seeding General) Grant awarded to Richard Siegert in 2016 16-AUT-024-CSG. The funding covered travel and accommodation costs for AS, RM and MB. MB is supported by a National Health and Medical

References (36)

  • W.F. Chaplin et al.

    Conceptions of states and traits: dimensional attributes with ideals as prototypes

    J. Personal. Social. Psychol.

    (1988)
  • L.M. Chavez et al.

    Applying generalizability theory methods to assess continuity and change on the Adolescent Quality of Life - Mental Health Scale (AQOL-MHS)

    Qual. Life Res.

    (2016)
  • D.A. Cole et al.

    Empirical and conceptual problems with longitudinal trait-state models: Introducing a trait-state-occasion model

    Psychol. Methods

    (2005)
  • J.A. Cranford et al.

    A procedure for evaluating sensitivity to within-person change: can mood measures in diary studies detect change reliably?

    Personal. Social. Psychol. Bull.

    (2006)
  • L.J. Cronbach et al.

    Theory of generalizability: a liberalization of reliability theory

    Br. J. Math. Stat. Psychol.

    (1963)
  • S.L. Crowley et al.

    The children's depression inventory: a comparison of generalizability and classical test theory analyses

    Educ. Psychol. Meas.

    (1994)
  • E. Gaudry et al.

    Validation of the state-trait distinction in anxiety research

    Multivar. Behav. Res.

    (1975)
  • M. Huisman

    Imputation of missing item responses: some simple techniques

    Qual. Quant.

    (2000)
  • Cited by (21)

    • Association between developmental dyslexia and anxiety/depressive symptoms among children in China: The chain mediating of time spent on homework and stress

      2022, Journal of Affective Disorders
      Citation Excerpt :

      It has ten items scored on a 0–2 Likert scale with five items reverse scored. Higher scores indicate more severe depressive symptoms (de la Vega et al., 2016; Paterson et al., 2018). The measure is validated for use in Chinese (Xie et al., 2020).

    View all citing articles on Scopus
    View full text