Performance of alternative COPD case-finding tools: a systematic review and meta-analysis

Elena Schnieders, Elyesa Ünal, Volker Winkler, Peter Dambach, Valérie R. Louis, Olaf Horstick, Florian Neuhann, Andreas Deckert
European Respiratory Review 2021 30: 200350; DOI: 10.1183/16000617.0350-2020

Abstract

Rationale Guidelines recommend pre-/post-bronchodilator spirometry for diagnosing COPD, but resource constraints limit the availability of spirometry in primary care in low- and middle-income countries. Although spirometry is the diagnostic gold standard, we shall assess alternative tools for settings without spirometry.

Methods A systematic literature review and meta-analysis was conducted, utilising Cochrane, CINAHL, Google Scholar, PubMed and Web of Science (search cut-off was May 01, 2020). Published studies comparing the accuracy of diagnostic tools for COPD with post-bronchodilator spirometry were considered. Studies without sensitivity/specificity data, without a separate validation sample and outside of primary care were excluded. Sensitivity, specificity and area under the curve (AUC) were assessed.

Results Of 7578 studies, 24 were included (14 635 participants). Hand devices yielded a larger AUC than questionnaires. The meta-analysis included 17 studies and the overall AUC of micro-spirometers (0.84, 95% CI 0.80–0.89) was larger when compared to the COPD population screener (COPD-PS) questionnaire (0.77, 95% CI 0.63–0.85) and the COPD diagnostic questionnaire (CDQ) (0.72, 95% CI 0.64–0.78). However, only the difference between micro-spirometers and the CDQ was significant.

Conclusions The CDQ and the COPD-PS questionnaire were approximately equally accurate tools. Questionnaires ensured testing of symptomatic patients, but micro-spirometers were more accurate. A combination could increase accuracy but was not evaluated in the meta-analysis.

Abstract

Micro-spirometry testing in symptomatic patients could enable COPD diagnosis in settings without spirometry. Questionnaires are valuable, as they are easy to implement and test for symptomatic patients. A combination could increase diagnostic accuracy. https://bit.ly/3opHzUA

Introduction

COPD is uncurable but is to a large degree preventable, with treatable symptoms. Defining manifestations are dyspnoea, chronic cough, high sputum production and frequent lower respiratory tract infections [1]. In 2017, COPD prevalence was about 300 million cases worldwide, with a relative increase of 5.9% from 1990 and it ranked as the 7th disease in the global list of years of life lost [2–4]. COPD caused 3.2 million deaths in 2017, with an increase of 17.5% from 2007 [2] and 90% of COPD-related deaths occur in low- and middle-income countries (LMICs) [5].

A growing elderly population and an ongoing exposure to risk factors are reasons why prevalence is anticipated to increase further [1]. In general, a minimum of 25% of smokers that have smoked for 25 years might acquire a clinically significant COPD [6]. Additionally, indoor air pollution through burning of biomass plays a large role, especially in LMICs [7]. Asthma, old age, HIV infection, a history or current tuberculosis (TB) infection and genetic dispositions like α1-antitrypsin deficiency can further contribute to the course of COPD [1]. The social and economic burden created by COPD is substantial. Not only does COPD lead to a high degree of mortality, but the morbidity when living with severe COPD is also consequential. The direct healthcare expenditure on acute lung infections (exacerbations) is significant, but the indirect costs should be considered as well. Severely affected people are no longer able to work and often need care, causing a serious problem particularly in LMICs [1].

The gold standard in diagnosing airflow obstruction is pre- and post-bronchodilator spirometry. Diagnosis of COPD is likely in symptomatic patients that have a post-bronchodilator forced expiratory volume in 1 s (FEV1)/forced vital capacity (FVC) ratio <0.7 or less than the lower limit of normal (LLN) [1]. It has been shown, when excluding patients with a history of asthma, that an early diagnosis of COPD is associated with a better outcome in terms of exacerbation rates, direct healthcare costs and mortality [8]. As lung function rates decrease the fastest in the mild stages of COPD, early diagnosis is important in order to slow this decline with smoking cessation therapy and medication [9]. It has been demonstrated that long-acting bronchodilators slow disease progression in the mild, early stages of COPD and increase quality of life [10, 11]. However, COPD is often undiagnosed or diagnosed late as resource constraints in primary care limit the availability of spirometry in many LMICs [12].

Guidelines recommend spirometry testing for diagnosing COPD in symptomatic patients but leave out what to do when no spirometry is available [13]. Alternative diagnostic tools have been developed and could be helpful in reducing the rate of underdiagnosis [14]. This work summarises all the alternative tools that were compared to post-bronchodilator spirometry in primary care populations and measures their quality. A work similar to this one was conducted in 2013, but many new tools have been developed since then. We aim to inform about best practice in COPD case-finding in a primary care setting without spirometry, ensuring that physicians around the world have easy access to current evidence when encountering a symptomatic patient likely to have COPD.

Methods

Search strategy and selection criteria

A systematic literature review was conducted to obtain an overview of the tools and to outline their sensitivity, specificity and area under the curve (AUC). Additionally, a meta-analysis was administered.

The PROSPERO registration number is: CRD42020197751 and this study adheres to the principles of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [15]. Database searches and screening of titles, abstracts, full text and reference lists, as well as quality assessment, were completed independently by E. Schnieders and E. Ünal. Any discrepancies were resolved between all the authors. The systematic literature search considered research available up until May 01, 2020 in Cochrane, CINAHL, PubMed and Web of Science. Furthermore, results in Google Scholar were screened in groups of 50 until no more relevant articles could be found. Consequently, the first 400 hits from Google Scholar were screened. Peer-reviewed articles in English, French, German and Spanish were evaluated and there was no limitation regarding past publication dates.

The search terms were derived from three different categories: COPD, diagnosis and primary healthcare. For each of these categories, possible alternative words and medical subject heading (MeSH) terms were determined (see supplementary table E1).

According to the PICO scheme, all studies conducted in a primary healthcare cohort that compared an alternative and reproducible diagnostic tool for COPD to gold standard post-bronchodilator spirometry and measured the sensitivity and specificity of the tool, were included. Protocols were excluded and only tools that focused solely on diagnosing COPD or airflow obstruction, defined as post-bronchodilator FEV1/FVC <0.7 or FEV1/FVC <LLN according to the Global Initiative for Chronic Obstructive Lung Disease (GOLD) criteria [1], were considered. The studies had to report the post-bronchodilator spirometry test results for every participant or every participant with pre-bronchodilator obstruction. Finally, studies that did not validate their new tool in a sample different to the one in which it was developed were excluded.

The meta-analysis included tools that were evaluated in at least five studies, as the statistical package does not recommend a small number of studies and issues a warning when less than five studies are used [16].

Data analysis

All identified results were imported into Endnote X9 (Clarivate Analytics, Philadelphia, PA, USA) and, after the removal of duplicates, titles and abstracts were screened with Rayyan [17]. The reference lists of all the included studies were then screened for further eligible studies. Data was then extracted using an Excel spreadsheet for the following categories: study setting, time of data collection, tool, cut-off, sample size, sample criteria, sample size calculation, spirometry criteria, definition of COPD, COPD patients, smokers, sensitivity, specificity and AUC. In cases where there was a lack of clarity regarding the use of post-bronchodilator spirometry, study authors were contacted.

Quality assessment of the included studies was performed with the QUADAS tool, evaluating applicability and the risk of bias for the categories “patient selection”, “index test”, “reference test” and “flow and timing” [18]. If a category was assessed as “unclear” for bias, then relevant information to answer the signalling questions was missing (for details see supplementary table E2). Biased studies are mentioned in the discussion but were not excluded from the analysis, given the low number of studies available.

The 95% confidence intervals (CIs) for the AUC values in the systematic review were calculated using the Wilson interval and the corresponding forest plot was constructed with the “Forestplot” package for R software (The R Project for Statistical Computing, www.r-project.org). Additionally, the Youden index was calculated [19].

The coupled forest plot of studies in the meta-analysis was created with Review Manager version 5.3 (The Nordic Cochrane Centre, Copenhagen, Denmark). The meta-analysis was conducted with R software using a bivariate model [20]. For this model, the function “reitsma” from the R-package “mada” was applied [16]. The “mada” package calculated the summary receiver operating characteristic (ROC) curve and therefore the overall AUC of each tool. As multiple thresholds of one tool were assessed, calculating the overall sensitivity and specificity was not recommended [21]. The 95% CIs for the summary AUC values were calculated using bootstrap with the R-package “dmetatools”, which also tested for statistically significant differences between the summary AUC values [22, 23]. The heterogeneity of the sensitivities and specificities of each tool was assessed graphically with a scatterplot and the summary ROC curve.

Results

Study selection

Altogether, 7578 potentially eligible papers were identified (see supplementary figure E1). After removing duplicates, 4823 articles were included in title and abstract screening, and 72 papers qualified for full-text reading. Of these, 48 were excluded, mainly due to violation of the spirometry inclusion criteria. Additionally, reference lists of eligible studies were searched; however, no additional studies were found.

General study characteristics

Ultimately, 24 studies dating from 2006 until 2020 were included [24–47]. Tables 1–3 show study criteria and tool performance results for diagnostic hand devices (i.e. micro-spirometers), questionnaires and a combination of hand devices and questionnaires, respectively. The studies were conducted in 18 different countries that are all considered high-income countries, apart from Venezuela, which is an upper middle-income country. Sample sizes ranged from 104 participants to 3234 participants [28, 34]. Study characteristics varied but most studies only included participants that were over 35 years of age, had no previous diagnosis of COPD and were current or ex-smokers.

View this table:
TABLE 1

Hand devices

View this table:
TABLE 2

Questionnaires

View this table:
TABLE 3

Questionnaires and hand devices

Quality assessment

The quality assessment determined that each diagnostic accuracy study was fully applicable to the research question (see supplementary table E3 and supplementary figure E2). Regarding the risk of bias, two studies were defined as being at “high risk” of bias in the category “patient selection” because they used a case-control design [35, 45]. In the “index test” category, nine studies were defined as “high risk” as they did not pre-specify a threshold for their tool. These studies were mainly micro-spirometer studies whose aim was the determination of the optimal threshold for FEV1/forced expiratory volume in 6 s (FEV6). Three of the nine “high risk” studies were evaluating new questionnaires whose scoring was not yet established [32, 41, 46]. The two studies defined as being “unclear” for bias in the “index test” category assessed multiple diagnostic tests, of which at least one did not have a pre-specified cut-off. The five studies marked as “unclear” for bias in the “reference test” category did not mention blinding of the interpreters of the reference test. In the category “flow and timing”, most studies were assessed to be of “high risk” of bias due to the exclusion of enrolled participants. The main reasons for this were poor quality of spirometry or incomplete data, with some reporting loss to follow-up as well. One study with a “high risk” of bias in the “flow and timing” category reported a large time interval between the index and reference test [33]. Studies assessed to be “unclear” for bias did not specify if any study participants were excluded.

Systematic review

The 24 included studies evaluated one Wheezometer, one peak expiratory flow (PEF) device, one discriminant function model, 11 different micro-spirometer thresholds, 11 unique questionnaires and two combinations of questionnaire and micro-spirometer [24–47]. A Wheezometer is a hand-held pulmonary sound analyser and a PEF device is a small hand tool that measures the PEF rate. Micro-spirometers are hand-held and mobile spirometers. The threshold for diagnosing COPD with a micro-spirometer is evaluated using the FEV1/FEV6 ratio instead of the FEV1/FVC ratio used in conventional spirometry. The micro-spirometers most evaluated in this review are the COPD-6 (Vitalograph Ltd., Ennis, County Clare, Ireland) and the Piko-6 (nSpire Health Inc., Longmont, CO, USA). Most questionnaires contained items such as age, smoking status and respiratory symptoms (e.g. wheezing, dyspnoea, coughing and the interference of symptoms with participants' daily lives).

A forest plot of the AUC of 38 tools, some of them identical, that were evaluated in 19 of the included studies is shown in figure 1. Results for hand devices, excluding the Wheezometer, showed higher AUC values on average than the questionnaires. A combination of questionnaire and micro-spirometer was evaluated in two studies. Only one of them determined the AUC [38]. The Youden index of the other study was calculated to be 0.69 [19, 26]. Both values were set in the same range as the hand device group excluding the Wheezometer. Eight studies evaluated more than one tool, which made inter-test comparison more meaningful [26–28, 30, 38, 43, 45, 47]. Six of these compared micro-spirometer use with questionnaires and micro-spirometers always showed a higher AUC or Youden index. Four studies compared multiple questionnaires and, in two of these four studies, the lung function questionnaire (LFQ) scored the highest AUC [27, 28]. The COPD diagnostic questionnaire (CDQ), also known as the international primary care airways guidelines (IPAG) questionnaire or the respiratory health screening questionnaire (RHSQ), yielded the highest AUC in the other two studies [30, 47]. In one of these four studies the LFQ and CDQ were compared directly. Here, the LFQ acquired the highest AUC (0.81), followed closely by the CDQ (0.80) and the COPD population screener (COPD-PS) questionnaire (0.79) [28]. The CDQ and the COPD-PS questionnaire were compared directly in three studies and the CDQ always obtained a slightly higher AUC [28, 30, 47]. Two cut-off values for the CDQ (≥16.5 and ≥19.5) were directly compared in four studies [29, 36, 41, 43]. In all studies, the cut-off value ≥19.5 yielded a higher Youden index in comparison to the threshold of ≥16.5.

FIGURE 1
FIGURE 1

Systematic review: forest plot of area under the curve (AUC) values with 95% confidence intervals (CIs) of hand devices (mainly the micro-spirometers COPD-6 and Piko-6), questionnaires, or both hand devices and questionnaires. The COPD-6 score was based on micro-spirometry results. PEF: peak expiratory flow; CDQ: COPD diagnostic questionnaire; COPD-PS: COPD population screener; VAFOSQ: veterans’ airflow obstruction screening questionnaire; PUMA: PUMA study questionnaire; LFQ: lung function questionnaire; 11-Q: 11-Q COPD screening questionnaire; SCSQ: Salzburg COPD-screening questionnaire; EGARPOC: EGARPOC study questionnaire; RHSQ: respiratory health screening questionnaire.

Meta-analysis

The meta-analysis included the CDQ and the COPD-PS questionnaire, as well as the micro-spirometers if they stated an FEV1/FEV6 cut-off. The three tools were assessed in 17 out of 24 studies. Only identical diagnostic tools that were evaluated at least five times were considered, but thresholds could differ. Some CDQ studies evaluated two different cut-offs but did not specify which one qualified as best for a clinical setting. As only one cut-off can be assessed with the bivariate model, the cut-off yielding the highest Youden index was applied for these studies, which was always ≥19.5. The thresholds of the micro-spirometers and the COPD-PS questionnaire used in the analysis were the ones specified to be the most efficient in their respective diagnostic accuracy studies or the ones with the highest Youden index. The following studies were included in the analysis: 8 CDQ studies (7115 participants), 5 COPD-PS questionnaire studies (4392 participants) and 10 micro-spirometer studies (2671 participants) evaluating pre-bronchodilator micro-spirometry [24–30, 33, 34, 36, 39, 41–45, 47]. Every participant in the CDQ group was over 35 years of age. Furthermore, with the exception of one study, every participant in the CDQ group had a history of smoking [26]. Apart from two studies, no participants with a previous diagnosis of COPD were included in the CDQ group and COPD prevalence spanned between 10.3% and 41% [26, 28, 30, 36]. Additionally, the eight CDQ studies were conducted in five different countries. Three out of five studies in the COPD-PS questionnaire group included participants that were specified to be >35 years old [28, 30, 47]. Study criteria regarding history of smoking and a previous diagnosis of COPD varied in the COPD-PS questionnaire group and COPD prevalence ranged between 10.7% and 45.6% [28, 45]. Furthermore, the five COPD-PS questionnaire studies were administered in four different countries. In all micro-spirometer studies but one, participants were specified as being >35 years old [45]. Eight out of the 10 micro-spirometer studies included only smokers and, apart from three studies, no participants with a previous diagnosis of COPD were included [24, 39, 45]. COPD prevalence in the micro-spirometer group ranged between 18.9% and 62% [24, 33]. Moreover, the 10 micro-spirometer studies were conducted in seven different countries. Supplementary figure E3 depicts the descriptive analysis of the sensitivity and specificity of each tool in each study in a coupled forest plot. Figure 2 displays a scatterplot of the sensitivities and specificities, as well as the summary ROC curve and the prediction region, for each tool resulting from the meta-analysis. The true sensitivity and specificity of a future tool lies with 95% confidence within this prediction region. Micro-spirometers obtained a greater overall AUC (0.84, 95% CI 0.80–0.89) than the questionnaires. The data analysis indicated that the COPD-PS questionnaire yielded a higher summary AUC (0.77, 95% CI 0.63–0.85) than the CDQ (0.72, 95% CI 0.64–0.78) (see supplementary figure E4). The differences between summary AUC values were tested for all three combinations of tools (see supplementary table E4). Only the summary AUC values of micro-spirometers and the CDQ differed significantly. A forest plot containing the AUC values of the included studies, as well as the summary estimates of the overall AUC values resulting from the bivariate analysis, can be found in supplementary figure E5.

FIGURE 2
FIGURE 2

Descriptive analysis and meta-analysis: scatterplot of sensitivities and specificities, as well as summary receiver operating characteristic (ROC) curves (including prediction region) for a) the COPD diagnostic questionnaire; b) the COPD population screener questionnaire; and c) micro-spirometers. AUC: area under the curve; CI: confidence interval; NA: not available.

Discussion

The aim of this study was to inform about current evidence concerning the optimal alternative diagnostic tool for COPD case-finding in primary care settings without spirometry.

Overall results

In the analysis of the systematic review, micro-spirometry almost always outperformed the analysed questionnaires. The combination of a hand device and a questionnaire was deemed as accurate at diagnosing COPD as the hand devices alone, given the similar AUC value and Youden index value. The meta-analysis determined that micro-spirometers are more accurate diagnostic tools than the CDQ and the COPD-PS questionnaire, although only the difference in summary AUC value compared to the CDQ was statistically significant. The COPD-PS questionnaire was a slightly more accurate diagnostic tool than the CDQ; however, the difference in summary AUC values was not statistically significant.

Micro-spirometers

Given the data analysed, micro-spirometers were a more accurate diagnostic tool for COPD in comparison to diagnostic questionnaires; however, only difference to the CDQ was significant. COPD diagnosis requires a symptomatic patient in addition to irreversible airflow obstruction. Of the 10 micro-spirometer studies, four were conducted in a patient cohort that was required to be symptomatic and almost all of the micro-spirometer study participants were required to have a history of smoking. Therefore, the conclusion that micro-spirometers are more accurate diagnostic tools is only valid for symptomatic patients or patients with a history of smoking and is not applicable to the general population. A systematic review and meta-analysis that answered a similar question, conducted by Haroon et al. [48], collected data up until December 2013. The eligibility criteria were different as only studies with participants that had no prior diagnosis of COPD were included. Furthermore, their spirometry criteria differed due to the acceptance of pre-bronchodilator spirometry as an adequate gold standard for the diagnosis of COPD. They concluded that micro-spirometers were more exact diagnostic tools for COPD in comparison to the CDQ, which was the only questionnaire included in their meta-analysis. The analysis in our work further supports their conclusions. Micro-spirometers are easier, quicker and cheaper to administer than conventional spirometry and could therefore be of great help in diagnosing symptomatic patients in resource-constrained, primary care settings.

Combination of tools

According to our systematic review, the combination of a micro-spirometer and a questionnaire was as capable in the diagnosis of COPD as micro-spirometers alone. COPD case-finding with a combination of a questionnaire and micro-spirometry has been deemed practical and has been recommended to be implemented into healthcare [49]. Apart from micro-spirometry, a PEF device can also be used in combination with a diagnostic questionnaire. A recently developed combination test in COPD diagnostic research is the COPD Assessment in Primary Care to Identify Undiagnosed Respiratory Disease and Exacerbation Risk questionnaire (CAPTURE) together with a PEF device. One study measured its sensitivity at 89.7% and its specificity at 78.1% [50]. Its Youden index value (0.68) is one of the highest in the range compared to our included studies. However, the CAPTURE+PEF evaluation study was excluded in this systematic review as not all study participants received post-bronchodilator spirometry. Soriano et al. [51] reported that a case-finding method comprised of the COPD-PS questionnaire and PEF led to a 90% reduction in the spirometry tests taken. A cross-sectional study from 2017 measured the sensitivity and specificity of respiratory questions in combination with PEF as 84% and 93%, respectively [52]. In summary, a combination of a questionnaire and a hand device can improve COPD diagnosis in primary care settings without spirometry. Diagnostic questionnaires can determine symptomatic patients and can therefore function as a pre-test to obstruction testing.

Questionnaires

Diagnostic questionnaires are still valuable tools, although micro-spirometers were assessed to be more accurate. This systematic review indicated that the LFQ was a slightly more robust diagnostic tool for COPD than the CDQ and the COPD-PS questionnaire [53–56]. According to the systematic review, the CDQ was slightly more accurate than the COPD-PS questionnaire. The meta-analysis demonstrated the contrary, although the difference was not statistically significant. Altogether the two questionnaires are approximately equally accurate in diagnosing COPD. The higher Youden index of the CDQ, with a threshold of ≥19.5 rather than a threshold of ≥16.5, suggested it might be the better option. A comparison of the CDQ and the COPD-PS questionnaire in a Japanese population determined that they were comparable in discriminating subjects with and without airflow limitation [57]. A validation study measured the sensitivity and specificity of the LFQ, with a threshold of ≤18, at 82.6% and 47.8%, respectively [58]. As in the two studies that were included here, this cut-off combines a high sensitivity with a low specificity. As stated above, COPD diagnosis requires a symptomatic patient. Few of the included questionnaire studies specified respiratory symptoms as an inclusion criterion, but the diagnostic questionnaires mentioned, namely the CDQ, LFQ and COPD-PS questionnaire, encompass questions about respiratory manifestations. In summary, diagnostic questionnaires for COPD can determine symptomatic patients and are easier to implement than hand devices, which is especially important in resource-limited settings.

Strengths and limitations

A strength of this work is that studies in four languages were assessed. Additionally, many new diagnostic tools were considered that have been developed since the last meta-analysis was conducted. Moreover, this review only included studies that used post-bronchodilator spirometry as a reference standard, which is defined as the gold standard by the GOLD guidelines [1].

As of 2016, the US Preventative Task Force undertook a systematic review resulting in it advising against screening for COPD in asymptomatic people, because there was no evidence of patient benefit [59, 60]. They concluded that more randomised and controlled studies in smokers, utilising the LFQ or CDQ as pre-screening questionnaires followed by micro-spirometry, are necessary. It is crucial to differentiate between COPD screening and case-finding. Screening is conducted in a population resembling the general population with a low prevalence of COPD. Case-finding is done in patients likely to have COPD due to respiratory symptoms or an exposure to noxious particles. Almost all included studies required participants to have a history of smoking, but potential non-smoking COPD patients and patients with exposure to biomass smoke might have been missed. Overall, only a few included studies required participants to be symptomatic, which is a limitation of this review. As we aimed to find an optimal case-finding tool, a realistic scenario would be if the tool were applied to symptomatic patients only. Unfortunately, there are few validation studies with this sample criterion and it was not feasible to only include studies with that requirement.

Given the strict eligibility criteria of this systematic review, many potentially valuable diagnostic tools for COPD were not analysed. Many recently developed questionnaires, like the veterans’ airflow obstruction screening questionnaire (VAFOSQ) and the PUMA study questionnaire, but also older examples like the LFQ, were eligible for the systematic review only, as they were solely tested twice [27, 28, 32, 37, 46]. Furthermore, only two combinations of a hand device and a questionnaire were assessed, and consequently they were not included in the meta-analysis. As such, definite statements about a combination of tools could not be made and overall the low number of included studies in the meta-analysis limits our results.

Another limitation of this meta-analysis is the heterogeneity of sensitivities and specificities (see figure 2). Heterogeneity can be assessed by the size of the prediction region and the distance of individual points from the summary ROC curve. According to these criteria, COPD-PS questionnaire studies show the most heterogeneity, followed by CDQ studies and finally micro-spirometer studies. Reasons for this could be that studies differed with regard to the patient's history of smoking, the inclusion of diagnosed COPD patients, the COPD prevalence and the country of conduct.

Regarding the risk of bias, it is possible that the diagnostic accuracy of some tools was evaluated optimistically, as can be observed in the “index test” category. “High risk” studies did not pre-specify a threshold for their tool. In the bivariate model only one cut-off per study could be assessed, therefore the optimal threshold resulting in the respective diagnostic accuracy study was chosen. As the optimal threshold was determined retrospectively, the stated quality measures might be lower in a different study sample. The exclusion of enrolled study participants from the analysis due to poor quality of spirometry could, for example, be another source of bias.

Using the AUC and the Youden index to search for the optimal diagnostic tool assumed that sensitivity and specificity were weighted as being equally important, which does not necessarily have to be the case. As a great proportion of COPD-related deaths occur in LMICs, a high specificity may be wanted to avoid unneeded strain on false positive participants and the healthcare system.

When available, spirometry should continue to be the first line of choice for COPD diagnosis, particularly considering that failure to make a correct diagnosis leads not only to underdiagnosis but also to overdiagnosis, with a related burden of disease and costs [61].

Suggestions for further research

The burden of COPD lies mainly in LMICs where spirometry is often not available in primary care settings. However, all 24 studies included in this review were, except for one, conducted in high-income countries. Therefore, further research regarding the applicability of these alternative diagnostic tools in low-resource settings is needed. Also, many questionnaires include respiratory symptoms and exposure to smoking, but not biomass fuel utilisation, which is an important risk factor in LMICs. Additionally, only two studies were included that evaluated a combination of a questionnaire and a hand device. As this combination could increase diagnostic accuracy, more research should be directed this way. When validating a COPD case-finding tool it should be ensured that only symptomatic patients are tested. Future studies should compare their tool to post-bronchodilator spirometry for every included participant or every included participant with pre-bronchodilator obstruction. Interpreters of index and reference tests should be blinded towards the results of the respective other test and exclusion of enrolled participants must be avoided.

Conclusion

We aimed to evaluate alternative diagnostic tools for COPD diagnosis in resource-constrained settings where no spirometry is available. In general, only symptomatic patients should be tested for COPD. The most accurate alternative diagnostic method to conventional spirometry, if sensitivity and specificity are equally weighted, is micro-spirometry. However, this was only shown in patients with a history of smoking and some symptomatic patient cohorts and the number of publications was low. Additionally, only the difference in accuracy to the CDQ was statistically significant. As they encompass questions about respiratory symptoms and are easy to implement, diagnostic questionnaires like the COPD-PS questionnaire and the CDQ are valuable diagnostic tools. Considering the systematic review, the meta-analysis, heterogeneity and CIs, both questionnaires are approximately equally accurate case-finding tools. They could be used as a test itself or as a pre-test for micro-spirometry or conventional spirometry. The combination of a questionnaire and micro-spirometry was deemed as accurate as micro-spirometry alone but was not evaluated in the meta-analysis. A combination could ensure obstruction testing of symptomatic patients only. When choosing an optimal diagnostic tool in primary care settings, a high specificity may be desired depending on resource availability. Alternative diagnostic tools for COPD can be of great help in resource-constrained, primary care settings. However, if post-bronchodilator spirometry is available, it should be used for final COPD diagnosis given its position as the gold standard.

Supplementary material

Supplementary Material

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary_tables_and figures ERR-0350-2020_Supplementary_tables_and_figures

Acknowledgements

The authors are affiliated to the Systematic Review Group of the Heidelberg Institute of Global Health. We obtained statistical consultation at the Institute for Medical Biometrics and Informatics at Heidelberg University, Germany.

Footnotes

  • Provenance: Submitted article, peer reviewed

  • This study is registered at https://www.crd.york.ac.uk/prospero/ with registration number: CRD42020197751.

  • This article has supplementary material available from err.ersjournals.com

  • Author contributions: A. Deckert initiated the study, conceived the research question and study design, and supervised the conduct of the study. E. Schnieders conducted the literature search and data extraction, as well as the quality assessment, analyses and the drafting of the paper. E. Ünal replicated the literature search and quality assessment. A. Deckert, F. Neuhann, V. Winkler, P. Dambach, V.R. Louis and O. Horstick monitored the analyses steps. All authors contributed to the writing of the paper.

  • Conflict of interest: E. Schnieders reports receiving a scholarship from the Heidelberg Graduate School of Global Health, which is funded by the Else-Kröner-Fresenius Stiftung.

  • Conflict of interest: E. Ünal has nothing to disclose.

  • Conflict of interest: V. Winkler has nothing to disclose.

  • Conflict of interest: P. Dambach has nothing to disclose.

  • Conflict of interest: V.R. Louis has nothing to disclose.

  • Conflict of interest: O. Horstick has nothing to disclose.

  • Conflict of interest: F. Neuhann has nothing to disclose.

  • Conflict of interest: A. Deckert has nothing to disclose.

  • Support statement: This publication was supported by the Heidelberg Graduate School of Global Health, which is funded by the Else-Kröner-Fresenius Stiftung. Funding information for this article has been deposited with the Crossref Funder Registry.

  • Received November 2, 2020.
  • Accepted January 20, 2021.
http://creativecommons.org/licenses/by-nc/4.0/

This version is distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0. For commercial reproduction rights and permissions contact permissions{at}ersnet.org

References

  1. ↵
    1. Global Initiative for Chronic Obstructive Lung Disease (GOLD)
    . Global Strategy for the Diagnosis, Management, and Prevention of COPD. 2020. Available from: http://goldcopd.org/
  2. ↵
    1. Roth GA,
    2. Abate D,
    3. Abate KH, et al.
    Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018; 392: 1736–1788. doi:10.1016/S0140-6736(18)32203-7
    OpenUrlCrossRefPubMed
    1. GBD 2017 Disease and Injury Incidence and Prevalence Collaborators
    . Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018; 392: 1789–1858. doi:10.1016/S0140-6736(18)32279-7
    OpenUrlCrossRefPubMed
  4. ↵
    1. Soriano JB,
    2. Kendrick PJ,
    3. Paulson KR, et al.
    Prevalence and attributable health burden of chronic respiratory diseases, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet Respir Med 2020; 8: 585–596. doi:10.1016/S2213-2600(20)30105-3
    OpenUrl
  5. ↵
    1. World Health Organization (WHO)
    . Chronic obstructive pulmonary disease (COPD). www.who.int/news-room/fact-sheets/detail/chronic-obstructive-pulmonary-disease-(copd) Date last updated: December 01, 2017. Date last accessed: October 08, 2020.
  6. ↵
    1. Løkke A,
    2. Lange P,
    3. Scharling H, et al.
    Developing COPD: a 25 year follow up study of the general population. Thorax 2006; 61: 935–939. doi:10.1136/thx.2006.062802
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Siddharthan T,
    2. Grigsby MR,
    3. Goodman D, et al.
    Association between household air pollution exposure and chronic obstructive pulmonary disease outcomes in 13 low- and middle-income country settings. Am J Respir Crit Care Med 2018; 197: 611–620. doi:10.1164/rccm.201709-1861OC
    OpenUrl
  8. ↵
    1. Larsson K,
    2. Janson C,
    3. Stallberg B, et al.
    Impact of COPD diagnosis timing on clinical and economic outcomes: the ARCTIC observational cohort study. Int J Chron Obstruct Pulmon Dis 2019; 14: 995–1008. doi:10.2147/COPD.S195382
    OpenUrl
  9. ↵
    1. Csikesz NG,
    2. Gartman EJ
    . New developments in the assessment of COPD: early diagnosis is key. Int J Chron Obstruct Pulmon Dis 2014; 9: 277–286. doi:10.2147/COPD.S46198
    OpenUrlPubMed
  10. ↵
    1. Decramer M,
    2. Celli B,
    3. Kesten S, et al.
    Effect of tiotropium on outcomes in patients with moderate chronic obstructive pulmonary disease (UPLIFT): a prespecified subgroup analysis of a randomised controlled trial. Lancet 2009; 374: 1171–1178. doi:10.1016/S0140-6736(09)61298-8
    OpenUrlCrossRefPubMed
  11. ↵
    1. Welte T,
    2. Vogelmeier C,
    3. Papi A
    . COPD: early diagnosis and treatment to slow disease progression. Int J Clin Pract 2015; 69: 336–349. doi:10.1111/ijcp.12522
    OpenUrlCrossRefPubMed
  12. ↵
    1. Kibirige D,
    2. Kampiire L,
    3. Atuhe D, et al.
    Access to affordable medicines and diagnostic tests for asthma and COPD in sub-Saharan Africa: the Ugandan perspective. BMC Pulm Med 2017; 17: 179. doi:10.1186/s12890-017-0527-y
    OpenUrl
  13. ↵
    1. National Institute for Health and Care Excellence (NICE)
    . Chronic obstructive pulmonary disease in over 16s: diagnosis and management (NICE guideline NG115). www.nice.org.uk/guidance/ng115/chapter/Recommendations#diagnosing-copd Date last updated: July 26, 2019. Date last accessed: September 28, 2020.
  14. ↵
    1. Diab N,
    2. Gershon AS,
    3. Sin DD, et al.
    Underdiagnosis and overdiagnosis of chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2018; 198: 1130–1139. doi:10.1164/rccm.201804-0621CI
    OpenUrl
  15. ↵
    1. Moher D,
    2. Liberati A,
    3. Tetzlaff J, et al.
    Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009; 62: 1006–1012. doi:10.1016/j.jclinepi.2009.06.005
    OpenUrlCrossRefPubMed
  16. ↵
    1. Doebler P,
    2. Holling H
    . Meta-analysis of diagnostic accuracy with mada. https://cran.r-project.org/web/packages/mada/vignettes/mada.pdf Date last updated: May 25, 2020. Date last accessed: October 08, 2020.
  17. ↵
    1. Ouzzani M,
    2. Hammady H,
    3. Fedorowicz Z, et al.
    Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016; 5: 210. doi: 10.1186/s13643-016-0384-4
    OpenUrlCrossRefPubMed
  18. ↵
    QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: 529–536. doi: 10.7326/0003-4819-155-8-201110180-00009
    OpenUrlCrossRefPubMed
  19. ↵
    1. Youden WJ
    . Index for rating diagnostic tests. Cancer 1950; 3: 32–35. doi:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3
    OpenUrlCrossRefPubMed
  20. ↵
    1. Gentleman R,
    2. Hornik K,
    3. Parmigiani G
    1. Schwarzer G,
    2. Carpenter JR,
    3. Rücker G
    . Analysis of diagnostic test accuracy studies. In: Gentleman R, Hornik K, Parmigiani G, eds. Meta-analysis with R. Springer International Publishing, 2015; pp. pp. 217–236. doi: 10.1007/978-3-319-21416-0_7
  21. ↵
    1. Deeks JJ,
    2. Bossuyt PM,
    3. Gatsonis C
    1. Macaskill P,
    2. Gatsonis C,
    3. Deeks JJ, et al.
    Chapter 10: analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, eds. Cochrane handbook for systematic reviews of diagnostic test accuracy (version 10). The Cochrane Collaboration, 2010; pp. 1–61. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/Chapter%2010%20-%20Version%201.0.pdf
  22. ↵
    1. Noma H
    . Dmetatools package: computational tools for meta-analysis of diagnostic accuracy test. 2020. https://github.com/nomahi/dmetatools/
  23. ↵
    1. Noma H,
    2. Matsushima Y
    . Confidence interval for the AUC of SROC curve and some related methods using bootstrap for meta-analysis of diagnostic accuracy studies. ArXiv 2020; preprint [https://arxiv.org/abs/2004.04339].
  24. ↵
    1. Dickens AP,
    2. Fitzmaurice DA,
    3. Adab P, et al.
    Accuracy of Vitalograph lung monitor as a screening test for COPD in primary care. NPJ Prim Care Respir Med 2020; 30: 2. doi:10.1038/s41533-019-0158-2
    OpenUrl
    1. Kim JK,
    2. Lee CM,
    3. Park JY, et al.
    Active case finding strategy for chronic obstructive pulmonary disease with handheld spirometry. Medicine 2016; 95: e5683. doi:10.1097/MD.0000000000005683
    OpenUrl
  26. ↵
    1. Sichletidis L,
    2. Spyratos D,
    3. Papaioannou M, et al.
    A combination of the IPAG questionnaire and PiKo-6® flow meter is a valuable screening tool for COPD in the primary care setting. Prim Care Respir J 2011; 20: 184–189. doi:10.4104/pcrj.2011.00038
    OpenUrlCrossRefPubMed
  27. ↵
    1. Sogbetun F,
    2. Eschenbacher WL,
    3. Welge JA, et al.
    A comparison of five surveys that identify individuals at risk for airflow obstruction and chronic obstructive pulmonary disease. Respir Med 2016; 120: 1–9. doi:10.1016/j.rmed.2016.09.010
    OpenUrl
  28. ↵
    1. Spyratos D,
    2. Haidich AB,
    3. Chloros D, et al.
    Comparison of three screening questionnaires for chronic obstructive pulmonary disease in the primary care. Respiration 2017; 93: 83–89. doi:10.1159/000453586
    OpenUrl
  29. ↵
    1. Stanley AJ,
    2. Hasan I,
    3. Crockett AJ, et al.
    COPD Diagnostic Questionnaire (CDQ) for selecting at-risk patients for spirometry: a cross-sectional study in Australian general practice. NPJ Prim Care Respir Med 2014; 24: 14024. doi:10.1038/npjpcrm.2014.24
    OpenUrl
  30. ↵
    1. Ronaldson SJ,
    2. Dyson L,
    3. Clark L, et al.
    Determining the optimal approach to identifying individuals with chronic obstructive pulmonary disease: the DOC study. J Eval Clin Pract 2018; 24: 487–495. doi:10.1111/jep.12896
    OpenUrl
    1. Weiss G,
    2. Steinacher I,
    3. Lamprecht B, et al.
    Development and validation of the Salzburg COPD-screening questionnaire (SCSQ): a questionnaire development and validation study. NPJ Prim Care Respir Med 2017; 27: 4. doi:10.1038/s41533-016-0005-7
    OpenUrl
  32. ↵
    1. Lopez Varela MV,
    2. Montes de Oca M,
    3. Rey A, et al.
    Development of a simple screening tool for opportunistic COPD case finding in primary care in Latin America: the PUMA study. Respirology 2016; 21: 1227–1234. doi:10.1111/resp.12834
    OpenUrl
  33. ↵
    1. Labor M,
    2. Vrbica Z,
    3. Gudelj I, et al.
    Diagnostic accuracy of a pocket screening spirometer in diagnosing chronic obstructive pulmonary disease in general practice: a cross sectional validation study using tertiary care as a reference. BMC Fam Pract 2016; 17: 112. doi:10.1186/s12875-016-0518-8
    OpenUrlPubMed
  34. ↵
    1. van den Bemt L,
    2. Wouters BC,
    3. Grootens J, et al.
    Diagnostic accuracy of pre-bronchodilator FEV1/FEV6 from microspirometry to detect airflow obstruction in primary care: a randomised cross-sectional study. NPJ Prim Care Respir Med 2014; 24: 14033. doi:10.1038/npjpcrm.2014.33
    OpenUrl
  35. ↵
    1. Cui J,
    2. Zhou Y,
    3. Tian J, et al.
    A discriminant function model as an alternative method to spirometry for COPD screening in primary care settings in China. J Thorac Dis 2012; 4: 594–600. doi:10.3978/j.issn.2072-1439.2012.11.06
    OpenUrlPubMed
  36. ↵
    1. Kotz D,
    2. Nelemans P,
    3. Van Schayck C, et al.
    External validation of a COPD diagnostic questionnaire. Eur Respir J 2008; 31: 298–303. doi:10.1183/09031936.00074307
    OpenUrlAbstract/FREE Full Text
  37. ↵
    1. Lopez Varela MV,
    2. Montes de Oca M,
    3. Wehrmeister FC, et al.
    External validation of the PUMA COPD diagnostic questionnaire in a general practice sample and the PLATINO study population. Int J Chron Obstruct Pulmon Dis 2019; 14: 1901–1911. doi:10.2147/COPD.S206250
    OpenUrl
  38. ↵
    1. Fujita M,
    2. Nagashima K,
    3. Takahashi S, et al.
    Handheld flow meter improves COPD detectability regardless of using a conventional questionnaire: a split-sample validation study. Respirology 2020; 25: 191–197. doi:10.1111/resp.13602
    OpenUrl
  39. ↵
    1. Thorn J,
    2. Tilling B,
    3. Lisspers K, et al.
    Improved prediction of COPD in at-risk patients using lung function pre-screening in primary care: a real-life study and cost-effectiveness analysis. Prim Care Respir J 2012; 21: 159–166. doi:10.4104/pcrj.2011.00104
    OpenUrlCrossRefPubMed
    1. Demirci H,
    2. Eniste K,
    3. Basaran EO, et al.
    A multicenter family practitioners’ research on chronic obstructive pulmonary disease screening using the COPD Assessment Test. Prim Health Care Res Dev 2017; 18: 603–607. doi:10.1017/S1463423617000408
    OpenUrl
  41. ↵
    1. Price DB,
    2. Tinkelman DG,
    3. Nordyke RJ, et al.
    Scoring system and clinical application of COPD diagnostic questionnaires. Chest 2006; 129: 1531–1539. doi:10.1378/chest.129.6.1531
    OpenUrlCrossRefPubMed
    1. Represas-Represas C,
    2. Fernandez-Villar A,
    3. Ruano-Ravina A, et al.
    Screening for chronic obstructive pulmonary disease: validity and reliability of a portable device in non-specialized healthcare settings. PLoS One 2016; 11: e0145571. doi:10.1371/journal.pone.0145571
    OpenUrlPubMed
  43. ↵
    1. Frith P,
    2. Crockett A,
    3. Beilby J, et al.
    Simplified COPD screening: validation of the PiKo-6 in primary care. Prim Care Respir J 2011; 20: 190–198. doi:10.4104/pcrj.2011.00040
    OpenUrlCrossRefPubMed
    1. Hidalgo Sierra V,
    2. Hernandez Mezquita MA,
    3. Palomo Cobos L, et al.
    Usefulness of the Piko-6 portable device for early COPD detection in primary care. Arch Bronconeumol 2018; 54: 460–466. doi:10.1016/j.arbres.2018.04.015
    OpenUrl
  45. ↵
    1. Miravitlles M,
    2. Llor C,
    3. Calvo E, et al.
    Validación de la versión traducida del Chronic Obstructive Pulmonary Disease-Population Screener (COPD-PS). Su utilidad y la del FEV1/FEV6 para el diagnóstico de enfermedad pulmonar obstructiva crónica [Validation of the Spanish version of the Chronic Obstructive Pulmonary Disease-Population Screener (COPD-PS). Its usefulness and that of FEV1/FEV6 for the diagnosis of COPD]. Med Clin (Barc) 2012; 139: 522–530. doi:10.1016/j.medcli.2011.06.022
    OpenUrlCrossRefPubMed
  46. ↵
    1. Sogbetun F,
    2. Eschenbacher WL,
    3. Welge JA, et al.
    Veterans airflow obstruction screening questionnaire: a survey to identify veterans with airflow obstruction. Chronic Obstr Pulm Dis 2016; 3: 705–715. doi:10.15326/jcopdf.3.4.2016.0128
    OpenUrl
  47. ↵
    1. Llordes M,
    2. Zurdo E,
    3. Jaen A, et al.
    Which is the best screening strategy for COPD among smokers in primary care? COPD 2017; 14: 43–51. doi:10.1080/15412555.2016.1239703
    OpenUrl
  48. ↵
    1. Haroon S,
    2. Jordan R,
    3. Takwoingi Y, et al.
    Diagnostic accuracy of screening tests for COPD: a systematic review and meta-analysis. BMJ Open 2015; 5: e008133. doi:10.1136/bmjopen-2015-008133
    OpenUrlAbstract/FREE Full Text
  49. ↵
    1. Frith PA
    . Detecting COPD using micro-spirometry and/or questionnaire. Respirology 2020; 25: 126–127. doi:10.1111/resp.13636
    OpenUrl
  50. ↵
    1. Martinez FJ,
    2. Mannino D,
    3. Leidy NK, et al.
    A new approach for identifying patients with undiagnosed chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2017; 195: 748–756. doi:10.1164/rccm.201603-0622OC
    OpenUrlPubMed
  51. ↵
    1. Soriano JB,
    2. Molina J,
    3. Miravitlles M
    . Combining case-finding methods for COPD in primary care: a large, two-stage design study. Int J Tuberc Lung Dis 2018; 22: 106–111. doi:10.5588/ijtld.17.0334
    OpenUrl
  52. ↵
    1. Thorat YT,
    2. Salvi SS,
    3. Kodgule RR
    . Peak flow meter with a questionnaire and mini-spirometer to help detect asthma and COPD in real-life clinical practice: a cross-sectional study. NPJ Prim Care Respir Med 2017; 27: 32. doi:10.1038/s41533-017-0036-8
    OpenUrl
  53. ↵
    1. Yawn BP,
    2. Mapel DW,
    3. Mannino DM, et al.
    Development of the Lung Function Questionnaire (LFQ) to identify airflow obstruction. Int J Chron Obstruct Pulmon Dis 2010; 5: 1–10. doi:10.2147/COPD.S7683
    OpenUrlPubMed
    1. Tinkelman DG,
    2. Price DB,
    3. Nordyke RJ, et al.
    Symptom-based questionnaire for differentiating COPD and asthma. Respiration 2006; 73: 296–305. doi:10.1159/000090141
    OpenUrlCrossRefPubMed
    1. Price DB,
    2. Tinkelman DG,
    3. Halbert R, et al.
    Symptom-based questionnaire for identifying COPD in smokers. Respiration 2006; 73: 285–295. doi:10.1159/000090142
    OpenUrlCrossRefPubMed
  56. ↵
    1. Martinez FJ,
    2. Raczek AE,
    3. Seifer FD, et al.
    Development and initial validation of a self-scored COPD Population Screener questionnaire (COPD-PS). COPD 2008; 5: 85–95. doi:10.1080/15412550801940721
    OpenUrlCrossRefPubMed
  57. ↵
    1. Tsukuya G,
    2. Samukawa T,
    3. Matsumoto K, et al.
    Comparison of the COPD Population Screener and International Primary Care Airway Group questionnaires in a general Japanese population: the Hisayama study. Int J Chron Obstruct Pulmon Dis 2016; 11: 1903–1909. doi:10.2147/COPD.S110429
    OpenUrl
  58. ↵
    1. Hanania NA,
    2. Mannino DM,
    3. Yawn BP, et al.
    Predicting risk of airflow obstruction in primary care: validation of the lung function questionnaire (LFQ). Respir Med 2010; 104: 1160–1170. doi:10.1016/j.rmed.2010.02.009
    OpenUrlCrossRefPubMed
  59. ↵
    1. Guirguis-Blake JM,
    2. Senger CA,
    3. Webber EM, et al.
    Screening for chronic obstructive pulmonary disease: a systematic evidence review for the US Preventive Services Task Force. Rockville, Agency for Healthcare Research and Quality (US), 2016.
  60. ↵
    1. Siu AL,
    2. Bibbins-Domingo K,
    3. Grossman DC, et al.
    Screening for chronic obstructive pulmonary disease: US Preventive Services Task Force recommendation statement. JAMA 2016; 315: 1372–1377. doi:10.1001/jama.2016.2638
    OpenUrlCrossRefPubMed
  61. ↵
    1. Ho T,
    2. Cusack RP,
    3. Chaudhary N, et al.
    Under- and over-diagnosis of COPD: a global perspective. Breathe 2019; 15: 24–35. doi:10.1183/20734735.0346-2018
    OpenUrlAbstract/FREE Full Text
Back to top
View this article with LENS
Email
Print
Citation Tools

Performance of alternative COPD case-finding tools: a systematic review and meta-analysis
Elena Schnieders, Elyesa Ünal, Volker Winkler, Peter Dambach, Valérie R. Louis, Olaf Horstick, Florian Neuhann, Andreas Deckert
European Respiratory Review Jun 2021, 30 (160) 200350; DOI: 10.1183/16000617.0350-2020
Reddit logo Technorati logo Twitter logo Connotea logo Facebook logo Mendeley logo
Full Text (PDF)

Jump To

Subjects