Abstract
Background Patient-reported outcome measures (PROMs) are recommended for capturing meaningful outcomes in clinical trials. The use of PROMs for children with acute lower respiratory infections (ALRIs) has not been systematically reported. We aimed to identify and characterise patient-reported outcomes and PROMs used in paediatric ALRI studies and summarise their measurement properties.
Methods Medline, Embase and Cochrane were searched (until April 2022). Studies that reported on patient-reported outcome (or measure) use or development and included subjects aged <18 years with ALRIs were included. Study, population and patient-reported outcome (or measure) characteristics were extracted.
Results Of 2793 articles identified, 18 met inclusion criteria, including 12 PROMs. Two disease-specific PROMs were used in settings in which they had been validated. The Canadian Acute Respiratory Illness and Flu Scale was the most frequently used disease-specific PROM (five studies). The EuroQol-Five Dimensions-Youth system was the most frequently used generic PROM (two studies). There was considerable heterogeneity in validation methods. The outcome measures identified in this review lack validation for young children and none involve sufficient content validity for use with First Nations children.
Conclusions There is an urgent need for PROM development that considers the populations in which the burden of ALRI predominates.
Abstract
This review found a lack of patient-reported outcome measures (PROMs) for paediatric acute lower respiratory infection (ALRI). PROM development that incorporates modern validation methods and considers those experiencing the greatest ALRI burden is needed. https://bit.ly/3iXVwfY
Introduction
Globally, acute lower respiratory infections (ALRIs) are the most common cause of morbidity and mortality in children under the age of 5 years [1]. Although ALRIs are common, have substantial health impacts and direct and indirect economic costs, critical knowledge gaps exist regarding the optimal management for children with respiratory infections [2]. For clinical trial results to be useful in addressing these gaps, the outcomes evaluated must be meaningful. A meaningful outcome captures how a patient feels, functions or survives, is acknowledged as meaningful to people with lived experience of the condition and has been evaluated in the populations intended for its use [3].
In high-income countries, the infrequent occurrence of objectively captured outcomes including hospitalisation or mortality presents challenges when considering them as end-points in clinical trials [4]. The selection of outcomes relevant to consumers improves the value of research and minimises research waste [5]. Patient (and parent/carer proxy) reported outcome measures (PROMs) have been suggested as a way to ensure patient-centred care for children with respiratory infections [6]. A patient-reported outcome (PRO) can be defined as an outcome reported from the perspective of a patient without interpretation by a third party [6]. Such outcomes include perceived symptoms, functional ability or experiences related to the health condition [7].
Despite experts advocating for the development and validation of PROMs, there is a lack of validated PROMs for use in ALRI trials [8, 9]. We aimed to systematically identify and characterise PROs and PROMs (self or proxy-report by parent/guardian) used for children with ALRIs and to summarise their properties of measurement.
Research methods
A PROSPERO registered (CRD42022308619) systematic review was conducted. Three primary databases (Medline, Embase and Cochrane) were searched from inception until 18 April 2022 through Ovid (see supplementary material 1). The search was limited to studies written in English and involving humans. Forward and backward citation searching of each included article was undertaken through Scopus and Google Scholar. Clinical trials and observational studies were eligible for inclusion if they 1) reported on the development and/or use of a PRO or PROM and 2) included children <18 years of age with ALRIs. Studies were excluded if they 1) exclusively evaluated chronic disease, 2) involved adults and children but did not separately report paediatric data or 3) utilised an unsuitable study format (e.g. conference proceeding).
Two authors (D.O. and M.J.B.) individually screened each article (title, keywords and abstract) in Covidence (https://covidence.org/). Both reviewers then screened the full texts of records remaining after the abstract screening. A third reviewer and subject matter expert (C.B.) was included to resolve discrepancies. Predetermined fields for data extraction included study type, characteristics of the population in which the PRO or PROM was developed and/or used and the PRO and PROM properties of measurement (supplementary material 2). The second author cross-checked extraction of the first 25% of the eligible articles ordered through Covidence. Validation data reporting on the quality (validity, reliability and responsiveness) of PROMs were appraised with reference to the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) criteria (appendix A) [10]. In considerations of content validity, the authors summarised the applicability of the participant demographics in PROM development studies to suggest evidence gaps. As the extracted data reported in this review were not individual outcomes, rather which PRO/PROMs were used and how they were used, a formal risk of bias assessment was not performed.
Results
PROs and PROMs
2793 studies were identified. Of these, 18 met the inclusion criteria. The PRISMA-S flow chart is depicted in figure 1. The study characteristics and outcome measures of the 18 identified articles are summarised in supplementary material 2. 12 PROMs were identified. Of these, three were validated for use in children with ALRIs. One generic and nine disease-specific PROMs were discovered; seven of which included activity and/or functional limitation items in addition to capturing disease-specific respiratory symptoms. Three PROs including the Canadian Acute Respiratory Illness and Flu Scale (CARIFS), the Gilead Respiratory Syncytial Virus (RSV) Caregiver Diary (GRCD) and the EuroQol-Five Dimensions-Youth (EQ-5D-Y) system were identified, which were independently ascertained as the outcome measures used in clinical trials [11–14].
PRISMA-S flow diagram reporting the systematic review results. ALRI: acute lower respiratory infection; PRO: patient-reported outcome.
The CARIFS was the most frequently used disease-specific PROM (n=5 studies) [7, 11, 12, 15, 16] and the EQ-5D-Y was the most frequently used generic PROM (n=2 studies) [14, 15]. Three validated PROMs were applied in settings in which their use had been validated, including the CARIFS, the GRCD and the EQ-5D-Y [7, 11, 13–17]. Six validated PROMs were used in settings in which they had not been validated. These included the CARIFS, the Bronchiolitis Caregiver Diary (BCD), the adapted Measure Yourself Medical Outcome Profile (MYMOP), the Asthma Symptom Diary, the Bronchitis Severity Scale (BSS) and the symptom questionnaire used by Kruizinga et al. [12, 18–24]. Two unvalidated PROMs were used, including the symptom diaries from Heinonen et al. [25], from Barratt et al. [26] and Matilla et al. [27].
The majority of identified studies were conducted after 2000 (n=15, 83.3%), including 10 studies (66.6%) published in the last 4 years. 10 were undertaken in Europe [14, 15, 18, 19, 25–27], two in Asia [16, 20] and four in North America [7, 11, 13, 17]. Two were conducted in more than one country [12, 21]. Children's age groups were well represented: 11 studies were conducted in children 0–14 months old; 11 articles in children 15–59 months old; and nine articles in children aged 60–216 months old (supplementary material 2). Studies were conducted in children with a broad spectrum of ALRIs, ranging from RSV-associated bronchiolitis (n=5) [13, 17, 18, 20, 21], influenza (n=6) [11, 12, 14, 15, 25, 27], bronchitis (n=1) [23], community-acquired pneumonia (n=2) [24, 26], ALRI (n=1) [19] and unspecified respiratory infections (n=4) [7, 16, 22].
The identified PROs and PROMs captured a variety of domains, including respiratory symptoms (12 uses), and the impact of illness on activity (10 uses) and behaviour (three uses). Of the identified PROs and PROMs, eight were composite measures of at least two of these three domains (table 1). The following reported respiratory symptoms were used: cough, rhinitis, vomiting, diarrhoea, headache, sore throat, muscle aches, fever, runny nose, shortness of breath, wheeze, feeling unwell, sputum production and noisy breathing. The PROs capturing the impact of illness on activity included the following: not sleeping well (child and parent), not playing well, needing extra care, being unable to get out of bed, mobility, usual activities, fitness, school absenteeism and days away from work (parents). Behavioural PROs captured included the following: poor appetite, irritability, low energy, crying more, clinginess, not being interested in what was going on, anxiety/depression, fussiness, and “not themselves”. Individual PROs not included as a composite PROM included “activity interference” and “general health”.
Characteristics of patient-reported outcome measures identified through systematic review
The two disease-specific PROMs that were both validated and used in the intended setting included the CARIFS and the GRCD. Both the CARIFS and the GRCD were designed as twice-daily parent-/guardian-reported outcome measures developed through systematic review and consultation with both subject experts and caregivers. These two PROMS each captured a range of respiratory symptoms, as well as the impact of illness on activity and behaviour (table 1). The CARIFS was validated for use in children aged <12 years with ALRIs (n=220) and the GRCD for use in infants with RSV aged <24 months (n=103).
Validation and properties of measurement of PROMs
Data regarding the properties of measurement of PROs/PROMs were identified for two of the 12 PROMs; these included the CARIFS and the GRCD (table 2). Definitions of the PROM properties of measurement can be found in appendix A.
Information reported on the quality (validity, reliability, and responsiveness) of identified patient-reported outcome measures (PROMs)
Both PROMs established adequate content validity through development utilising literature review and subsequent paediatrician and caregiver focus groups for item generation and reduction. Significant discriminant validity was found for the CARIFS in terms of distinguishing children who were likely to require further healthcare visits (including to their general practitioner or to the hospital) subsequent to their initial presentation [7]. The GRCD reported mixed discriminant validity between those rated as “more ill” by alternative severity scores [13]. Item-level ANOVA testing was performed for the GRCD (higher means for patients rated as more ill in 84.2% of 57 ANOVAs), which found that only a few held statistically significant mean differences (14.0% of 57 ANOVAs) [13]. The CARIFS found high correlation between clinician (rS=0.36) and parental assessment (rS=0.52), demonstrating acceptable concurrent validity [7]. The GRCD items described moderate to strong correlations with caregiver reports (r=0.30–0.63); however, less with clinician reported outcomes (r=−0.02–0.34) [13]. In terms of reliability, intra-class correlations for the CARIFS were adequate (a day 2 Cronbach's alpha score of 0.808) and item-level test–retest reliability assessment for the GRCD found that most items had an acceptable correlation (17 of 19 items) [7, 13]. Both PROMs demonstrated satisfactory internal consistency (Cronbach's alpha for CARIFS: 0.89; GRCD: 0.78–0.94) [7, 13]. Regarding PROM responsiveness, both demonstrated decreases in scores from day 0 until recovery [7, 13].
Three of the 18 studies describing the use of PROMs did not reference a development or validation study [25–27]. The adapted MYMOP PROM did not report validation for children [19]. The BCD was originally developed and validated for asthma patients and adapted for post-acute RSV bronchiolitis [18, 20, 21]. The symptom diary card utilised by Wat et al. [22] was originally developed to capture symptom recovery during asthma exacerbations. The symptom questionnaire used by Kruizinga et al. [24] was validated for use in adults (>21 years old). The BSS, utilised by Tran et al. [23], was validated for clinician reporting of bronchitis symptoms in all age groups accompanied by patient consultation.
Discussion
This is the first review to systematically identify and characterise PROMs for use in studies in children with ALRIs. There are three important findings from this study. Firstly, there are few ALRI PROMs validated for use in children. Importantly, none have been specifically developed or validated for use in First Nations children, who disproportionately bear the highest burden of disease [28]. Secondly, only seven of 18 articles identified in this study used PROMs in a setting for which the outcome measure had been validated in and intended for use. Consequently, PROMs that have been validated for use in adult populations are currently being translated for use in paediatric settings, which may not capture meaningful outcomes nor provide easily interpretable or translatable results. Finally, the measurement properties for PROMs were inconsistently reported across different studies. This may be partly attributable to the fact that consensus recommendations for the development and validation of PROMs only emerged in 2017 [10].
While a high-quality outcome measure should be validated in accordance with published standards, the validation studies captured in this review use a range of different methodologies to capture validity, reliability and responsiveness [10]. The two agreed methods for PROM psychometric validation include classical test theory (CTT), often using factor analysis and Cronbach's alpha to refine items, and modern psychometric methods, such as item response theory (IRT), which selects items through evaluating the probability of an item response given a respondent's individual latent trait. Both the CARIFS and GRCD were validated using the CTT approach, which assumes a common precision estimate among all respondents. While it has been argued that modern psychometric methods can overcome this limitation of using CTT methodology, there are properties of PROM measurement that can only be assessed with CTT, such as test–retest reliability. The COSMIN guidelines recommend both CTT and IRT approaches when performing a validation study [10].
Both the CARIFS and the GRCD demonstrate adequate content validity through item generation and refinement involving those with lived experience of the disease. To meet acceptable convergent validity (see appendix A), correlations for the CARIFS (r=0.52) and GRCD (r=0.30–0.63) were adequate between constructs that are related yet dissimilar, such as the “parental global assessment” used for the CARIFS or the “caregiver-reported assessment” used in the GRCD. The analysis of groups known to differ regarding RSV severity utilised for the GRCD resulted in limited evidence of the GRCD's discriminative ability. In contrast, the survival analysis from the CARIFS depicted adequate discriminative validity to significantly distinguish between children with different healthcare utilisation.
The intra-class reliability for the CARIFS (≥0.7) and the test–retest reliability from the GRCD (≥0.7) were adequately demonstrated. The CARIFS and the GRCD demonstrated adequate internal consistency (0.808 and 0.78–0.94, respectively). Responsiveness of PROMs are often evaluated through moderate to large effect sizes (reported for the GRCD) and response means (reported for both the CARIFS and the GRCD). Both the CARIFS and the GRCD are missing important properties such as measurement error and minimally important difference.
As demonstrated by the CARIFS and the GRCD, PROM development is centred around the patient consultation. As such, a PROM evaluation study is often population, setting and context specific. For example, a PROM developed to measure outcomes of paediatric ALRI in high-income settings like Australia may fail to capture a patient's experience living in a low–middle income setting. From an Australian perspective, as the burden of ALRIs are overrepresented in First Nations communities, PROM development needs to consider its utility and relevance to the patients and families that would benefit the most. Despite having been validated, when PROMs are used in settings that have not been previously validated, for example, the Asthma Symptom Diary [22] or the BSS [23], the constructs of interest may not be accurately measured. The collaborative resolve, time and funding required to develop and validate PROMs within specific populations and circumstances in which it is purposed provides a barrier to progression that is not traditionally considered when designing clinical trials.
A potential limitation of the review could arise from the complexity in how ALRIs are defined in medical research. This was addressed through a rigorously designed systematic search strategy developed in conjunction with domain experts. The two-reviewer screening, selection and extraction process likely reduced selection bias. Previous reviews [8, 9] have noted the difficulty in searching for historic PROMs as the terminology was not as established, finding that retrospective backwards citation searching was required to identify older PROMs.
Points for clinical practice and questions for future research
• Despite experts highlighting the importance of PROs in trials, there is a lack of PROMs for use within paediatric ALRI studies.
• Through designing PROMs for populations where ALRI burden predominates, trials can better measure the effect of interventions on ALRI burden that are meaningful to families.
Conclusion
As few PROMs exist that are acceptable for use in studies involving children with ALRIs and none are specific for use in First Nations children, PROMs are commonly used or adapted for use outside the settings in which they have undergone validation assessment. PROM development should be informed by consultation with end-users of clinical trial evidence, including subject matter experts and those with lived experience of the disease under study. There is a pressing need to develop a validated PROM for use in ALRI studies in children as one measure of treatment success in order to ensure the value of the research that is conducted in this population.
Supplementary material
Supplementary Material
Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.
Supplementary material ERR-0229-2022.supplement
Acknowledgements
In addition to the authors of this manuscript, the study was supported by the assistance of The University of Western Australia staff, especially librarian Kylie Black, who provided expert advice for the review search strategy.
Footnotes
Provenance: Submitted article, peer reviewed.
Conflict of interest: The authors have no conflicts of interest to disclose.
Support statement: No funding was required for this study. C.C. Blyth is supported by an NHMRC Investigator award (1173163); C. McLeod is supported by a Raine Clinician Fellowship.
- Received November 23, 2022.
- Accepted January 17, 2023.
- Copyright ©The authors 2023
This version is distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0. For commercial reproduction rights and permissions contact permissions{at}ersnet.org
References
