Risk of malignancy in pulmonary nodules: A validation study of four prediction models

doi:10.1016/j.lungcan.2015.03.018

Lung Cancer

Volume 89, Issue 1, July 2015, Pages 27-30

https://doi.org/10.1016/j.lungcan.2015.03.018 Get rights and content

Highlights

•
The Mayo and Brock models performed well in predicting nodule malignant risk in clinical practice.
•
The Veterans Association model had the lowest accuracy of the models assessed.
•
The Brock model had the highest AUC for sub-centimetre pulmonary nodules.
•
In patients undergoing PET–CT, the model by Herder et al. had the highest accuracy.

Abstract

Objectives

Clinical prediction models assess the likelihood of malignancy in pulmonary nodules detected by computed tomography (CT). This study aimed to validate four such models in a UK population of patients with pulmonary nodules. Three models used clinical and CT characteristics to predict risk (Mayo Clinic, Veterans Association, Brock University) with a fourth model (Herder et al. [4]) additionally incorporating ¹⁸Fluorine-Fluorodeoxyglucose (FDG) avidity on positron emission tomography–computed tomography (PET–CT).

Materials and methods

The likelihood of malignancy was calculated for patients with pulmonary nodules (4–30 mm diameter) and data used to calculate the area under the receiver operating characteristic curve (AUC) for each model. The models were used in a restricted cohort of patients based on each model's exclusion criteria and in the total cohort of all patients.

Results

Two hundred and forty-four patients were studied, of whom 139 underwent FDG PET–CT. Ninety-nine (40.6%) patients were subsequently confirmed to have malignant nodules (33.2% primary lung cancer, 7.4% metastatic disease). The Mayo and Brock models performed similarly (AUC 0.895 and 0.902 respectively) and both were significantly better than the Veterans Association model (AUC 0.735, p < 0.001 and p = 0.002 respectively). In patients undergoing FDG PET–CT, the Herder model had significantly higher accuracy than the other three models (AUC 0.924). When the models were tested on all patients in the cohort (i.e. including those outside the original model inclusion criteria) AUC values were reduced, yet remained high especially for the Herder model (AUC 0.916). For sub-centimetre nodules, AUC values for the Mayo and Brock models were 0.788 and 0.852 respectively.

Conclusions

The Mayo and Brock models showed good accuracy for determining likelihood of malignancy in nodules detected on CT scan. In patients undergoing FDG PET–CT for nodule evaluation, the highest accuracy was seen for the model described by Herder et al. incorporating FDG avidity.

Introduction

Pulmonary nodules are a common incidental finding on thoracic computed tomography (CT) scans, and the investigation and surveillance of such nodules comprises a significant workload for clinical teams. The continuing rise in CT use together with possible uptake of CT screening for lung cancer is expected to increase the number of nodules detected over coming years.

Several quantitative prediction models using clinical and radiological criteria have been developed to assist clinicians in discriminating malignant from benign nodules. Three models incorporate clinical and CT nodule characteristics (Mayo Clinic [1], Veterans Association (VA) [2], and Brock University [3]) with a fourth model (Herder et al. [4]) additionally incorporating ¹⁸Fluorine-Fluorodeoxyglucose (FDG) avidity on positron emission tomography–computed tomography (PET–CT).

Although widely quoted, the Brock University model and the model described by Herder et al. have not been externally validated in any study, and none of these models have been validated in a UK population. The aim of this study was therefore to compare the performance of these models in a population of patients recruited from a UK teaching hospital.

Section snippets

Methods

Patients with pulmonary nodules were retrospectively identified from the lung cancer multi-disciplinary team (MDT) meeting and a nodule follow-up clinic between 2008 and 2013. Patients were eligible for inclusion if the diameter of their dominant nodule was 4–30 mm. Patients with up to 5 nodules other than the index nodule were also included. Patients were excluded from the study if the CT showed consolidation, pleural effusion, lymphadenopathy, or metastatic disease. Due to local

Results

In total, 244 patients were eligible for the study. The median age of the patients was 69 years (range 32–94 years), 50% were male and 76.2% were current or ex-smokers. Thirty six patients (14.8%) had a history of extra-thoracic cancer diagnosed within the last 5 years, and 11 patients (4.5%) had an extra-thoracic cancer diagnosed more than 5 years previously. Seven patients (2.9%) had a previous history of lung cancer. One hundred and ninety eight patients (81.1%) had a solitary nodule, and

Discussion

To our knowledge, this is the first time that the Herder and Brock models have been validated in any published series.

Before using clinical prediction models to assess risk, clinicians should be certain that the models are valid in their patient populations. The prevalence of malignancy is likely to vary with the demography of the local population, and by geography relating to the prevalence of granulomatous disease. The models differed in their recruitment methods; in the Mayo study patients

Funding

None.

Conflict of interest

None.

Acknowledgments

None.

References (4)

M.K. Gould et al.
A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules
Chest
(2007)
G.J. Herder et al.
Clinical prediction model to characterize pulmonary nodules: validation and added value of 18F-fluorodeoxyglucose positron emission tomography
Chest
(2005)

There are more references available in the full text version of this article.

Cited by (127)

Lung Cancer resection in the absence of pre-operative histology: The Accuracy of Multidisciplinary Team Consensus
2023, European Journal of Surgical Oncology
Lung resection remains the gold-standard of treatment for non-small cell lung cancer (NSCLC). British Thoracic Society (BTS) guidelines recommends the pursuit of pre-operative histological diagnosis and staging where possible. In the absence of pre-operative histology, surgical treatment can be offered in conjunction with multidisciplinary team (MDT) and patient consensus. We undertook a single-centre analysis of the accuracy of the lung cancer MDT in recommending surgical resection for those with suspected lung malignancy in the absence of pre-operative histological diagnosis over a six-year period.
A retrospective review was performed of patients undergoing any pulmonary oncological resection with therapeutic intent without pre-operative histological diagnosis at the recommendation of the lung cancer MDT in our unit between May 2016 and July 2022.
270 consecutive patients underwent lung resection for a lung nodule of indeterminate significance. This accounted for 45% of the oncological resections performed over this period. The mean age of the cohort was 67.9 years, and 47.4% were male. Overall, 10% of resected specimens (n = 27) were benign on final histopathology. 93% of those undergoing a lobectomy received a malignant diagnosis. Across the study cohort, surgical resection was well tolerated with a low complication rate.
Lung cancer resection in the absence of pre-operative histological diagnosis is feasible in a select patient cohort. This approach requires an experienced multi-disciplinary team and careful patient counselling. Our study demonstrates this adapted approach to be a pragmatic solution to the management of indeterminate pulmonary nodules in centres where biopsy is not routinely available due to existing constraints on the health system.
Differences in VA and Non-VA Pulmonary Nodules: All Evaluations Are not Created Equal
2023, Clinical Lung Cancer
Indeterminate pulmonary nodules present a common challenge for clinicians who must recommend surveillance or intervention based on an assessed risk of malignancy.
In this cohort study, patients presenting for indeterminate pulmonary nodule evaluation were enrolled at sites participating in the Colorado SPORE in Lung Cancer. They were followed prospectively and included for analysis if they had a definitive malignant diagnosis, benign diagnosis, or radiographic resolution or stability of their nodule for > 2 years.
Patients evaluated at the Veterans Affairs (VA) and non-VA sites were equally as likely to have a malignant diagnosis (48%). The VA cohort represented a higher-risk group than the non-VA cohort regarding smoking history and chronic obstructive pulmonary disease (COPD). There were more squamous cell carcinoma diagnoses among VA malignant nodules (25% vs. 10%) and a later stage at diagnosis among VA patients. Discrimination and calibration of risk calculators produced estimates that were wide-ranging and different when comparing between risk score calculators as well as between VA/non-VA cohorts. Application of current American College of Chest Physicians guidelines to our groups could have resulted in inappropriate resection of 12% of benign nodules.
Comparison of VA with non-VA patients shows important differences in underlying risk, histology of malignant nodules, and stage at diagnosis. This study highlights the challenge in applying risk calculators to a clinical setting, as the model discrimination and calibration were variable between calculators and between our higher-risk VA and lower-risk non-VA groups.
Microabstract: Risk stratification and management of indeterminate pulmonary nodules (IPNs) is a common clinical problem. In this prospective cohort study of 282 patients with IPNs from Veterans Affairs (VA) and non-VA sites, we found differences in patient and nodule characteristics, histology and diagnostic stage, and risk calculator performance. Our findings highlight challenges and shortcomings of current IPN management guidelines and tools.
Design and methodological considerations for biomarker discovery and validation in the Integrative Analysis of Lung Cancer Etiology and Risk (INTEGRAL) Program
2023, Annals of Epidemiology
The Integrative Analysis of Lung Cancer Etiology and Risk (INTEGRAL) program is an NCI-funded initiative with an objective to develop tools to optimize low-dose CT (LDCT) lung cancer screening. Here, we describe the rationale and design for the Risk Biomarker and Nodule Malignancy projects within INTEGRAL.
The overarching goal of these projects is to systematically investigate circulating protein markers to include on a panel for use (i) pre-LDCT, to identify people likely to benefit from screening, and (ii) post-LDCT, to differentiate benign versus malignant nodules. To identify informative proteins, the Risk Biomarker project measured 1161 proteins in a nested-case control study within 2 prospective cohorts (n = 252 lung cancer cases and 252 controls) and replicated associations for a subset of proteins in 4 cohorts (n = 479 cases and 479 controls). Eligible participants had a current or former history of smoking and cases were diagnosed up to 3 years following blood draw. The Nodule Malignancy project measured 1078 proteins among participants with a heavy smoking history within four LDCT screening studies (n = 425 cases diagnosed up to 5 years following blood draw, 430 benign-nodule controls, and 398 nodule-free controls).
The INTEGRAL panel will enable absolute quantification of 21 proteins. We will evaluate its performance in the Risk Biomarker project using a case-cohort study including 14 cohorts (n = 1696 cases and 2926 subcohort representatives), and in the Nodule Malignancy project within five LDCT screening studies (n = 675 cases, 680 benign-nodule controls, and 648 nodule-free controls). Future progress to advance lung cancer early detection biomarkers will require carefully designed validation, translational, and comparative studies.
A radiomics-based decision support tool improves lung cancer diagnosis in combination with the Herder score in large lung nodules
2022, eBioMedicine
Large lung nodules (≥15 mm) have the highest risk of malignancy, and may exhibit important differences in phenotypic or clinical characteristics to their smaller counterparts. Existing risk models do not stratify large nodules well. We aimed to develop and validate an integrated segmentation and classification pipeline, incorporating deep-learning and traditional radiomics, to classify large lung nodules according to cancer risk.
502 patients from five U.K. centres were recruited to the large-nodule arm of the retrospective LIBRA study between July 2020 and April 2022. 838 CT scans were used for model development, split into training and test sets (70% and 30% respectively). An nnUNet model was trained to automate lung nodule segmentation. A radiomics signature was developed to classify nodules according to malignancy risk. Performance of the radiomics model, termed the large-nodule radiomics predictive vector (LN-RPV), was compared to three radiologists and the Brock and Herder scores.
499 patients had technically evaluable scans (mean age 69 ± 11, 257 men, 242 women). In the test set of 252 scans, the nnUNet achieved a DICE score of 0.86, and the LN-RPV achieved an AUC of 0.83 (95% CI 0.77–0.88) for malignancy classification. Performance was higher than the median radiologist (AUC 0.75 [95% CI 0.70–0.81], DeLong p = 0.03). LN-RPV was robust to auto-segmentation (ICC 0.94). For baseline solid nodules in the test set (117 patients), LN-RPV had an AUC of 0.87 (95% CI 0.80–0.93) compared to 0.67 (95% CI 0.55–0.76, DeLong p = 0.002) for the Brock score and 0.83 (95% CI 0.75–0.90, DeLong p = 0.4) for the Herder score. In the international external test set (n = 151), LN-RPV maintained an AUC of 0.75 (95% CI 0.63–0.85). 18 out of 22 (82%) malignant nodules in the Herder 10–70% category in the test set were identified as high risk by the decision-support tool, and may have been referred for earlier intervention.
The model accurately segments and classifies large lung nodules, and may improve upon existing clinical models.
This project represents independent research funded by: 1) Royal Marsden Partners Cancer Alliance, 2) the Royal Marsden Cancer Charity, 3) the National Institute for Health Research (NIHR) Biomedical Research Centre at the Royal Marsden NHS Foundation Trust and The Institute of Cancer Research, London, 4) the National Institute for Health Research (NIHR) Biomedical Research Centre at Imperial College London, 5) Cancer Research UK (C309/A31316).
Predicting lung nodules malignancy
2022, Pulmonology
It is critical to developing an accurate method for differentiating between malignant and benign solitary pulmonary nodules. This study aimed was to establish a predicting model of lung nodules malignancy in a real-world setting.
The authors retrospectively analysed the clinical and computed tomography (CT) data of 121 patients with lung nodules, submitted to percutaneous CT-guided transthoracic biopsy, between 2014 and 2015. Multiple logistic regression was used to screen independent predictors for malignancy and to establish a clinical prediction model to evaluate the probability of malignancy.
From a total of 121 patients, 75 (62%) were men and with a mean age of 64.7 years old. Multivariate logistic regression analysis identified six independent predictors of malignancy: age, gender, smoking status, current extra-pulmonary cancer, air bronchogram and nodule size (p<0.05). The area under the curve (AUC) was 0.8573.
The prediction model established in this study can be used to assess the probability of malignancy in the Portuguese population, thereby providing help for the diagnosis of lung nodules and the selection of follow-up interventions.
Pulmonary nodule malignancy probability: a diagnostic accuracy meta-analysis of the Mayo model
2022, Clinical Radiology
Citation Excerpt :
Less than 50% of the patient population was a current or former smoker in the remaining 23 populations 17,19,21–24,27,29,35,36,41,43,49,51,52,54 with a lower AUC (AUC 0.695, 95% CI: 0.660-0.730, p<0.001, I2=88.19%); however, the CIs of the two groups overlap. Regarding study region, studies conducted in Asia17,19,21–24,27–29,31,33–36,43,49–54 (China, Japan, or Korea) had a pooled AUC of 0.699 (95% CI: 0.669–0.729, p<0.001, I2=91.20%) whereas the AUC for studies conducted in continents other than Asia4,16,18,20,25,26,32,37–42,45–48 was 0.769 (95% CI: 0.732–0.806, p<0.001, I2 = 95.72%; Fig 4). A subgroup on prevalence of malignant cases was also performed.
To quantify the overall diagnostic accuracy of the Mayo model in predicting the probability of malignancy of solitary pulmonary nodules (SPNs) by carrying out a systematic review and meta-analysis.
PubMed, Scopus, and references of relevant articles were searched systematically from inception up to 21 September 2021, to identify observational cohort studies, which reported on the diagnostic accuracy of the Mayo model in predicting the probability of malignancy in SPNs. The primary outcome was the pooled area under the receiver operating characteristic (ROC) curve (AUC). Secondary outcomes were sensitivity, specificity, and diagnostic odds ratio (DOR). The meta-analysis was registered with PROSPERO.
Forty studies involving 46,694 patients were included. The overall pooled AUC was 0.730 (95% confidence interval [CI]: 0.704–0.755), while sensitivity and specificity were 0.78 (95% CI: 0.67–0.86) and 0.68 (95% CI: 0.57–0.77), respectively. Subgroup analyses revealed that the model had a better performance among datasets with increased prevalence of current/ex-smokers (AUC 0.762, 95% CI: 0.713–0.810, versus AUC 0.695, 95% CI: 0.660–0.730 and among studies conducted in continents other than Asia (AUC 0.769, 95% CI: 0.732–0.806 versus AUC 0.699, 95% CI: 0.669–0.729, p<0.001, I2=91.20%)
The Mayo model seems a fair diagnostic tool; however, the synthesised evidence informs clinicians and researchers that it may be more applicable on certain patient populations, such as smokers and non-Asians.

View all citing articles on Scopus

View full text

Risk of malignancy in pulmonary nodules: A validation study of four prediction models

Highlights

Abstract

Objectives

Materials and methods

Results

Conclusions

Introduction

Section snippets

Methods

Results

Discussion

Funding

Conflict of interest

Acknowledgments

Chest

Chest