Advertisement

Performance of a deep learning-based lung nodule detection system as an alternative reader in a Chinese lung cancer screening program

  • Author Footnotes
    1 These authors contributed equally to this work.
    Xiaonan Cui
    Correspondence
    Corresponding author at: Tianjin Medical University Cancer Institute and Hospital, Huan-Hu-Xi Road, Ti-Yuan-Bei, He Xi District, Tianjin 300060, China.
    Footnotes
    1 These authors contributed equally to this work.
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China

    University of Groningen, University Medical Center Groningen, Department of Radiology, Groningen, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 These authors contributed equally to this work.
    Sunyi Zheng
    Footnotes
    1 These authors contributed equally to this work.
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands

    University of Groningen, University Medical Center Groningen, Machine Learning Lab, Data Science Center in Health, Groningen, The Netherlands
    Search for articles by this author
  • Marjolein A. Heuvelmans
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Epidemiology, Groningen, The Netherlands
    Search for articles by this author
  • Yihui Du
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Epidemiology, Groningen, The Netherlands
    Search for articles by this author
  • Grigory Sidorenkov
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Epidemiology, Groningen, The Netherlands
    Search for articles by this author
  • Shuxuan Fan
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Yanju Li
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Yongsheng Xie
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Zhongyuan Zhu
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Monique D. Dorrius
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Radiology, Groningen, The Netherlands
    Search for articles by this author
  • Yingru Zhao
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Raymond N. J. Veldhuis
    Affiliations
    University of Twente, Faculty of Electrical Engineering Mathematics and Computer Science, The Netherlands
    Search for articles by this author
  • Geertruida H. de Bock
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Epidemiology, Groningen, The Netherlands
    Search for articles by this author
  • Matthijs Oudkerk
    Affiliations
    University of Groningen, Faculty of Medical Sciences, The Netherlands
    Search for articles by this author
  • Peter M. A. van Ooijen
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands

    University of Groningen, University Medical Center Groningen, Machine Learning Lab, Data Science Center in Health, Groningen, The Netherlands
    Search for articles by this author
  • Rozemarijn Vliegenthart
    Affiliations
    University of Groningen, University Medical Center Groningen, Department of Radiology, Groningen, The Netherlands
    Search for articles by this author
  • Zhaoxiang Ye
    Affiliations
    Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Centre of Cancer, Key Laboratory of Cancer Prevention and Therapy, Department of Radiology, Tianjin, The People's Republic of China
    Search for articles by this author
  • Author Footnotes
    1 These authors contributed equally to this work.
Published:November 24, 2021DOI:https://doi.org/10.1016/j.ejrad.2021.110068

      Abstract

      Objective

      To evaluate the performance of a deep learning-based computer-aided detection (DL-CAD) system in a Chinese low-dose CT (LDCT) lung cancer screening program.

      Materials and Methods

      One-hundred-and-eighty individuals with a lung nodule on their baseline LDCT lung cancer screening scan were randomly mixed with screenees without nodules in a 1:1 ratio (total: 360 individuals). All scans were assessed by double reading and subsequently processed by an academic DL-CAD system. The findings of double reading and the DL-CAD system were then evaluated by two senior radiologists to derive the reference standard. The detection performance was evaluated by the Free Response Operating Characteristic curve, sensitivity and false-positive (FP) rate. The senior radiologists categorized nodules according to nodule diameter, type (solid, part-solid, non-solid) and Lung-RADS.

      Results

      The reference standard consisted of 262 nodules ≥ 4 mm in 196 individuals; 359 findings were considered false positives. The DL-CAD system achieved a sensitivity of 90.1% with 1.0 FP/scan for detection of lung nodules regardless of size or type, whereas double reading had a sensitivity of 76.0% with 0.04 FP/scan (P = 0.001). The sensitivity for detection of nodules ≥ 4 - ≤ 6 mm was significantly higher with DL-CAD than with double reading (86.3% vs. 58.9% respectively; P = 0.001). Sixty-three nodules were only identified by the DL-CAD system, and 27 nodules only found by double reading. The DL-CAD system reached similar performance compared to double reading in Lung-RADS 3 (94.3% vs. 90.0%, P = 0.549) and Lung-RADS 4 nodules (100.0% vs. 97.0%, P = 1.000), but showed a higher sensitivity in Lung-RADS 2 (86.2% vs. 65.4%, P < 0.001).

      Conclusions

      The DL-CAD system can accurately detect pulmonary nodules on LDCT, with an acceptable false-positive rate of 1 nodule per scan and has higher detection performance than double reading. This DL-CAD system may assist radiologists in nodule detection in LDCT lung cancer screening.

      Keywords

      To read this article in full you will need to make a payment

      References

      1. W. Chen, R. Zheng, P.D. Baade, S. Zhang, H. Zeng, F. Bray, A. Jemal, X.Q. Yu, J. He, Cancer statistics in China, 2015, CA: a cancer journal for clinicians 66(2) (2016) 115-32.

        • Cui X.
        • Han D.
        • Heuvelmans M.A.
        • Du Y.
        • Zhao Y.
        • Zhang L.
        • Groen H.J.M.
        • de Bock G.H.
        • Dorrius M.D.
        • Oudkerk M.
        • Vliegenthart R.
        • Ye Z.
        Clinical characteristics and work-up of small to intermediate-sized pulmonary nodules in a Chinese dedicated cancer hospital.
        Cancer Biol Med. 2020; 17: 199-207
        • Oudkerk M.
        • Devaraj A.
        • Vliegenthart R.
        • Henzler T.
        • Prosch H.
        • Heussel C.P.
        • Bastarrika G.
        • Sverzellati N.
        • Mascalchi M.
        • Delorme S.
        • Baldwin D.R.
        • Callister M.E.
        • Becker N.
        • Heuvelmans M.A.
        • Rzyman W.
        • Infante M.V.
        • Pastorino U.
        • Pedersen J.H.
        • Paci E.
        • Duffy S.W.
        • de Koning H.
        • Field J.K.
        European position statement on lung cancer screening.
        The Lancet Oncology. 2017; 18: e754-e766
      2. W.J. Yang, F.F. Qian, J.J. Teng, H.M. Wang, C. Manegold, L.R. Pilz, W. Voigt, Y.W. Zhang, J.D. Ye, Q.H. Chen, B.H. Han, A.M.E.T.S. Collaborative, Community-based lung cancer screening with low-dose CT in China: Results of the baseline screening, Lung Cancer 117 (2018) 20-26.

        • Zhou Q.
        • Fan Y.
        • Wu N.
        • Huang Y.
        • Wang Y.
        • Li L.
        • Liu J.
        • Wang X.
        • Li W.
        • Qiao Y.
        Demonstration program of population-based lung cancer screening in C hina: Rationale and study design.
        Thoracic cancer. 2014; 5: 197-203
      3. N.L.S.T.R. Team, Reduced lung-cancer mortality with low-dose computed tomographic screening, N. Engl. J. Med. 365(5) (2011) 395-409.

        • Kauczor Hans-Ulrich
        • Baird Anne-Marie
        • Blum Torsten Gerriet
        • Bonomo Lorenzo
        • Bostantzoglou Clementine
        • Burghuber Otto
        • Čepická Blanka
        • Comanescu Alina
        • Couraud Sébastien
        • Devaraj Anand
        • Jespersen Vagn
        • Morozov Sergey
        • Nardi Agmon Inbar
        • Peled Nir
        • Powell Pippa
        • Prosch Helmut
        • Ravara Sofia
        • Rawlinson Janette
        • Revel Marie-Pierre
        • Silva Mario
        • Snoeckx Annemiek
        • van Ginneken Bram
        • van Meerbeeck Jan P.
        • Vardavas Constantine
        • von Stackelberg Oyunbileg
        • Gaga Mina
        ESR/ERS statement paper on lung cancer screening.
        Eur. Radiol. 2020; 55: 1900506https://doi.org/10.1183/13993003.00506-201910.1183/13993003.00506-2019.Supp110.1183/13993003.00506-2019.Shareable1
        • Du Y.
        • Zhao Y.
        • Sidorenkov G.
        • de Bock G.H.
        • Cui X.
        • Huang Y.
        • Dorrius M.D.
        • Rook M.
        • Groen H.J.
        • Heuvelmans M.A.
        Methods of computed tomography screening and management of lung cancer in Tianjin: design of a population-based cohort study.
        Cancer biology & medicine. 2019; 16: 181
        • Van Iersel C.A.
        • De Koning H.J.
        • Draisma G.
        • Mali W.P.
        • Scholten E.T.
        • Nackaerts K.
        • Prokop M.
        • Habbema J.D.F.
        • Oudkerk M.
        • Van Klaveren R.J.
        Risk-based selection from the general population in a screening trial: selection criteria, recruitment and power for the Dutch-Belgian randomised lung cancer multi-slice CT screening trial (NELSON).
        Int. J. Cancer. 2007; 120: 868-874
        • Oudkerk Matthijs
        • Liu ShiYuan
        • Heuvelmans Marjolein A.
        • Walter Joan E.
        • Field John K.
        Lung cancer LDCT screening and mortality reduction—evidence, pitfalls and future perspectives.
        Nature Reviews Clinical Oncology. 2021; 18: 135-151
        • Lopez Torres E.
        • Fiorina E.
        • Pennazio F.
        • Peroni C.
        • Saletta M.
        • Camarlinghi N.
        • Fantacci M.E.
        • Cerello P.
        Large scale validation of the M5L lung CAD on heterogeneous CT datasets.
        Med. Phys. 2015; 42: 1477-1489
        • Roberts Heidi C.
        • Patsios Demetris
        • Kucharczyk Michael
        • Paul Narinder
        • Roberts Timothy P.
        The utility of computer-aided detection (CAD) for lung cancer screening using low-dose CT.
        International Congress Series, Elsevier. 2005; 1281: 1137-1142
      4. N.L.S.T.R. Team, Results of initial low-dose computed tomographic screening for lung cancer, N. Engl. J. Med. 368(21) (2013) 1980-1991.

        • Zhao Y.
        • de Bock G.H.
        • Vliegenthart R.
        • van Klaveren R.J.
        • Wang Y.
        • Bogoni L.
        • de Jong P.A.
        • Mali W.P.
        • van Ooijen P.M.
        • Oudkerk M.
        Performance of computer-aided detection of pulmonary nodules in low-dose CT: comparison with double reading by nodule volume.
        Eur. Radiol. 2012; 22: 2076-2084
      5. K.N. Jeon, J.M. Goo, C.H. Lee, Y. Lee, J.Y. Choo, N.K. Lee, M.-S. Shim, I.S. Lee, K.G. Kim, D.S.J.I.r. Gierada, Computer-aided nodule detection and volumetry to reduce variability between radiologists in the interpretation of lung nodules at low-dose screening CT, 47(8) (2012) 457.

      6. E.J. Hwang, C.M.J.K.J.o.R. Park, Clinical Implementation of Deep Learning in Thoracic Radiology: Potential Applications and Challenges, 21(5) (2020) 511.

        • Ye X.
        • Lin X.
        • Dehmeshki J.
        • Slabaugh G.
        • Beddoe G.
        Shape-based computer-aided detection of lung nodules in thoracic CT images.
        IEEE Transactions on Biomedical Engineering. 2009; 56: 1810-1820
      7. K. Murphy, A. Schilham, H. Gietema, M. Prokop, B. van Ginneken, Automated detection of pulmonary nodules from low-dose computed tomography scans using a two-stage classification system based on local image features, Medical Imaging 2007: Computer-Aided Diagnosis, International Society for Optics and Photonics, 2007, p. 651410.

        • Liu J.
        • Cao L.
        • Akin O.
        • Tian Y.
        3DFPN-HS $$^ $$: 3D Feature Pyramid Network Based High Sensitivity and Specificity Pulmonary Nodule Detection.
        International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. 2019; : 513-521
        • Armato III, S.G.
        • McLennan G.
        • Bidaut L.
        • McNitt-Gray M.F.
        • Meyer C.R.
        • Reeves A.P.
        • Zhao B.
        • Aberle D.R.
        • Henschke C.I.
        • Hoffman E.A.
        The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans.
        Med. Phys. 2011; 38: 915-931
      8. The Cancer Imaging Archive. TCIA collections. . http://www.cancerimagingarchive.net/. 02.08.2016).

        • Tan M.
        • Deklerck R.
        • Cornelis J.
        • Jansen B.
        Phased searching with NEAT in a time-scaled framework: experiments on a computer-aided detection system for lung nodules.
        Artif. Intell. Med. 2013; 59: 157-167
      9. B. Van Ginneken, A.A. Setio, C. Jacobs, F. Ciompi, Off-the-shelf convolutional neural network features for pulmonary nodule detection in computed tomography scans, 2015 IEEE 12th International symposium on biomedical imaging (ISBI), IEEE, 2015, pp. 286-289.

      10. P.-P. Ypsilantis, G. Montana, Recurrent convolutional networks for pulmonary nodule detection in CT imaging, arXiv preprint arXiv:1609.09143 (2016).

        • Setio A.A.A.
        • Ciompi F.
        • Litjens G.
        • Gerke P.
        • Jacobs C.
        • Van Riel S.J.
        • Wille M.M.W.
        • Naqibullah M.
        • Sánchez C.I.
        • van Ginneken B.
        Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks.
        IEEE Trans. Med. Imaging. 2016; 35: 1160-1169
        • Firmino M.
        • Angelo G.
        • Morais H.
        • Dantas M.R.
        • Valentim R.
        Computer-aided detection (CADe) and diagnosis (CADx) system for lung cancer with likelihood of malignancy.
        Biomedical engineering online. 2016; 15: 2
        • Dou Qi
        • Chen Hao
        • Yu Lequan
        • Qin Jing
        • Heng Pheng-Ann
        Multilevel contextual 3-D CNNs for false positive reduction in pulmonary nodule detection.
        IEEE Trans. Biomed. Eng. 2017; 64: 1558-1567
        • Setio Arnaud Arindra Adiyoso
        • Traverso Alberto
        • de Bel Thomas
        • Berens Moira S.N.
        • Bogaard Cas van den
        • Cerello Piergiorgio
        • Chen Hao
        • Dou Qi
        • Fantacci Maria Evelina
        • Geurts Bram
        • Gugten Robbert van der
        • Heng Pheng Ann
        • Jansen Bart
        • de Kaste Michael M.J.
        • Kotov Valentin
        • Lin Jack Yu-Hung
        • Manders Jeroen T.M.C.
        • Sóñora-Mengana Alexander
        • García-Naranjo Juan Carlos
        • Papavasileiou Evgenia
        • Prokop Mathias
        • Saletta Marco
        • Schaefer-Prokop Cornelia M
        • Scholten Ernst T.
        • Scholten Luuk
        • Snoeren Miranda M.
        • Torres Ernesto Lopez
        • Vandemeulebroucke Jef
        • Walasek Nicole
        • Zuidhof Guido C.A.
        • Ginneken Bram van
        • Jacobs Colin
        Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge.
        Med. Image Anal. 2017; 42: 1-13
        • Jiang Hongyang
        • Ma He
        • Qian Wei
        • Gao Mengdi
        • Li Yan
        An automatic detection system of lung nodule based on multigroup patch-based deep learning network.
        IEEE journal of biomedical and health informatics. 2018; 22: 1227-1237
        • Zhang J.
        • Xia Y.
        • Zeng H.
        • Zhang Y.
        NODULe: Combining constrained multi-scale LoG filters with densely dilated 3D deep convolutional neural network for pulmonary nodule detection.
        Neurocomputing. 2018; 317: 159-167
        • Zheng Sunyi
        • Guo Jiapan
        • Cui Xiaonan
        • Veldhuis Raymond N.J.
        • Oudkerk Matthijs
        • van Ooijen Peter M.A.
        Automatic Pulmonary Nodule Detection in CT Scans Using Convolutional Neural Networks Based on Maximum Intensity Projection.
        IEEE Trans Med Imaging. 2020; 39: 797-805
      11. Lung CT Screening Reporting & Data System (Lung-RADS), 2019. https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads.

        • Bunch P.C.
        • Hamilton J.F.
        • Sanderson G.K.
        • Simmons A.H.
        A free response approach to the measurement and characterization of radiographic observer performance, Application of Optical Instrumentation in Medicine VI.
        International Society for Optics and Photonics. 1977; : 124-135
        • Bandos A.I.
        • Rockette H.E.
        • Song T.
        • Gur D.
        Area under the free-response ROC curve (FROC) and a related summary index.
        Biometrics. 2009; 65: 247-256
        • Niemeijer Meindert
        • Loog Marco
        • Abràmoff Michael David
        • Viergever Max A
        • Prokop Mathias
        • van Ginneken Bram
        On combining computer-aided detection systems.
        IEEE Trans. Med. Imaging. 2011; 30: 215-223
        • Jankowski A.
        • Martinelli T.
        • Timsit J.-F.
        • Brambilla C.
        • Thony F.
        • Coulomb M.
        • Ferretti G.
        Pulmonary nodule detection on MDCT images: evaluation of diagnostic performance using thin axial images, maximum intensity projections, and computer-assisted detection.
        European radiology. 2007; 17: 3148-3156
        • Ardila Diego
        • Kiraly Atilla P.
        • Bharadwaj Sujeeth
        • Choi Bokyung
        • Reicher Joshua J.
        • Peng Lily
        • Tse Daniel
        • Etemadi Mozziyar
        • Ye Wenxing
        • Corrado Greg
        • Naidich David P.
        • Shetty Shravya
        End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. 2019; 25: 954-961
        • Jacobs C.
        • van Rikxoort E.M.
        • Murphy K.
        • Prokop M.
        • Schaefer-Prokop C.M.
        • van Ginneken B.
        Computer-aided detection of pulmonary nodules: a comparative study using the public LIDC/IDRI database.
        Eur. Radiol. 2016; 26: 2139-2147
        • Li L.
        • Liu Z.
        • Huang H.
        • Lin M.
        • Luo D.
        Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: Comparison with the performance of double reading by radiologists.
        Thoracic cancer. 2019; 10: 183-192
        • Filho Antonio Oseas de Carvalho
        • Silva Aristófanes Corrêa
        • de Paiva Anselmo Cardoso
        • Nunes Rodolfo Acatauassú
        • Gattass Marcelo
        3D shape analysis to reduce false positives for lung nodule detection systems.
        Med Biol Eng Comput. 2017; 55: 1199-1213
        • Cascio D.
        • Magro R.
        • Fauci F.
        • Iacomi M.
        • Raso G.
        Automatic detection of lung nodules in CT datasets based on stable 3D mass–spring models. 2012; 42: 1098-1109
        • Wormanns D.
        • Fiebich M.
        • Saidi M.
        • Diederich S.
        • Heindel W.
        Automatic detection of pulmonary nodules at spiral CT: clinical application of a computer-aided diagnosis system.
        Eur. Radiol. 2002; 12: 1052-1057
        • Brown M.S.
        • Goldin J.G.
        • Suh R.D.
        • McNitt-Gray M.F.
        • Sayre J.W.
        • Aberle D.R.
        Lung micronodules: automated method for detection at thin-section CT—initial experience.
        Radiology. 2003; 226: 256-262
        • Armato S.G.
        • Li F.
        • Giger M.L.
        • MacMahon H.
        • Sone S.
        • Doi K.
        Lung cancer: performance of automated lung nodule detection applied to cancers missed in a CT screening program.
        Radiology. 2002; 225: 685-692
        • Al Mohammad B.
        • Brennan P.C.
        • Mello-Thoms C.
        A review of lung cancer screening and the role of computer-aided detection.
        Clin. Radiol. 2017; 72: 433-442