Combined cumulative sum (CUSUM) and chronological environmental analysis as a tool to improve the learning environment for linear-probe endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) trainees: a pilot study
© The Author(s). 2017
Received: 3 August 2016
Accepted: 31 January 2017
Published: 7 February 2017
Cumulative sum (CUSUM) analysis can be used to continuously monitor the performance of an individual or process and detect deviations from a preset or standard level of achievement. However, no previous study has evaluated the utility of CUSUM analysis in facilitating timely environmental assessment and interventions to improve performance of linear-probe endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA). The aim of this study was to evaluate the usefulness of combined CUSUM and chronological environmental analysis as a tool to improve the learning environment for EBUS-TBNA trainees.
This study was an observational chart review. To determine if performance was acceptable, CUSUM analysis was used to track procedural outcomes of trainees in EBUS-TBNA. To investigate chronological changes in the learning environment, multivariate logistic regression analysis was used to compare several indices before and after time points when significant changes occurred in proficiency.
Presence of an additional attending bronchoscopist was inversely associated with nonproficiency (odds ratio, 0.117; 95% confidence interval, 0–0.749; P = 0.019). Other factors, including presence of an on-site cytopathologist and dose of sedatives used, were not significantly associated with duration of nonproficiency.
Combined CUSUM and chronological environmental analysis may be useful in hastening interventions that improve performance of EBUS-TBNA.
KeywordsCUSUM analysis Quality improvement Endobronchial ultrasound-guided transbronchial needle aspiration EBUS-TBNA CUSUM analysis Quality improvement
Cumulative sum (CUSUM) analysis is a method of monitoring the performance of an individual or process and detect deviations from a preset or standard level of achievement [1, 2]. It has been used in medical disciplines, particularly in monitoring and characterizing trainee learning curves. In theory, CUSUM analysis is an ideal tool to improve the quality of medical procedures, as it can alert observers to performance declines or improvements among workers, thus facilitating timely assessment of the learning environment. However, no previous study has examined the utility of combining CUSUM analysis with assessment of the learning environment for medical procedures.
Linear-probe endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is a bronchoscopic technique that uses ultrasound to visualize and guide sampling of mediastinal and hilar lymph nodes [3, 4]. Kemp et al. used CUSUM analysis to plot learning curves for EBUS-TBNA and found that the curves varied widely among bronchoscopists . Differences in learning speed among bronchoscopists at the same institution are mostly attributable to variation in their individual characteristics. However, if a bronchoscopist has an unexpected important change in proficiency in CUSUM analysis, investigating the learning environment and current performance characteristics of the bronchoscopist could help in implementing appropriate early interventions.
This study investigated whether CUSUM analysis could be used to monitor individual and institutional proficiency in linear EBUS-TBNA and improve the quality of the learning environment for EBUS-TBNA. CUSUM analysis was used to monitor EBUS-TBNA performance of bronchoscopists during training and investigate the relationship between change in performance and changes in the characteristics of the learning environment in the bronchoscopy suite.
Study design and setting
Linear-probe EBUS-TBNA bronchoscopy has been performed since March 2008 at our institution, the Saint Louis University Hospital. All attending bronchoscopists had more than 5 years of experience in bronchoscopy without EBUS and had received less than 1 week of EBUS-TBNA training before EBUS-TBNA was initiated at our center. We analyzed data from 131 patients evaluated from March 2008 to September 2010 by 3 bronchoscopists (bronchoscopist A: 47 patients with 84 lymph nodes sampled; bronchoscopist B: 50 patients with 89 lymph nodes sample; and bronchoscopist C: 34 patients with 49 lymph nodes sampled). All procedures utilized 22G needles.
Charts and procedure notes were examined to collect information on patient demographics, EBUS-TBNA indications, the procedures (date, operator(s), sampled lymph nodes, complications), final cytology results of transbronchial needle aspiration for each node, pathology results for mediastinoscopy, CT-guided lymph node biopsy, video-assisted thoracic surgery and thoracotomy, findings of subsequent imaging studies, and final clinical diagnosis.
Information on bronchoscopy suite conditions that was available retrospectively included the name of the main attending bronchoscopist (resident and fellow physicians were not included in the analysis), the names of attending bronchoscopists who assisted the main bronchoscopist, the doses of sedatives and opiates used during each procedure, and the presence of cytopathologists for rapid on-site evaluation of specimens. Two bronchoscopy nurses were present for all procedures. This observational chart review was approved by the Institutional Review Board at Saint Louis University.
Kemp et al. defined success on EBUS-TBNA as a true positive or true negative for malignancy, and failure as a false positive or false negative for malignancy. However, a procedure that successfully samples a lymph node but yields a false negative should not be defined as operator failure, as it represents a limitation of the EBUS-TBNA technique itself rather than a lack of operator skill. We believe that procedure failure should instead be defined as absence of lymphocytes, malignant cells, or granuloma in a specimen. In other words, a procedure is successful when the lymph node is adequately visualized and sampled, as determined by the presence of diagnostic cells or lymphocytes in the sample.
The primary outcome was sampling failure or success. Each sampled lymph node was used as the unit for determining success or failure. A lymph node sampling was defined as successful if 1 or more cytology slides showed lymphocytes, granulomas, or malignant cells.
To investigate chronological changes in learning environment, multivariate logistic regression analysis was used to compare each index before and after time points when significant proficiency changes occurred. To evaluate factors associated with a worsening interval, a multivariate-adjusted logistic regression model was constructed by including the presence of additional attending bronchoscopists or on-site cytopathologists, sedative dose (midazolam dose of ≥10 mg vs a lower dose), and opiate dose (fentanyl dose of ≥300 μg vs a lower dose). For measures of association, 95% confidence intervals were computed, and statistical significance was set at a P value of <0.05 (2-tailed). STATA 12.0 software was used for all analyses.
In the analysis of institutional EBUS-TBNA proficiency, we divided the study period into 4 intervals, namely, the intervals between cases 1 and 26 (learning interval), between cases 27 and 51 (proficiency interval), between cases 52 and 73 (worsening interval), and after case 73 (proficiency interval). To identify factors associated with success rate, we compared indices during the worsening interval (between cases 52 and 73) with those during the other intervals. A multivariate-adjusted logistic regression model was used to analyze the effects of the presence of additional attending bronchoscopists and on-site cytopathologists and higher doses of sedatives (≥10 mg of midazolam) and opiates (≥300 microgram of fentanyl). Comparison of the worsening interval with the other intervals revealed that presence of an additional attending bronchoscopist was inversely associated with nonproficiency (odds ratio [OR], 0.117; 95% confidence interval [CI], 0–0.749; P = 0.019). Other factors were not significantly associated with nonproficiency, including presence of on-site cytopathologists (OR, 0.365; 95% CI, 0.0369-1.835; P = 0.322), use of a midazolam dose of ≥10 mg (OR, 0.847; 95% CI, 0.167-3.512; P = 0.999, n = 38), and use of a fentanyl dose of ≥200 μg (OR, 2.786; 95% CI, 0.562-15.471; P = 0.2588, n = 22). The numbers of procedures performed by operators A and B did not significantly differ in any interval. Operator C performed fewer procedures than the other operators during intervals 1 (P < 0.016), 3 (worsening interval) (P < 0.045), and 4 (P < 0.075).
Summary of results
A previous study using CUSUM analysis of the EBUS-TBNA learning curve found that the learning rate varied, even among experienced bronchoscopists . In addition, our data suggest that chronological analysis and comparisons of variables before and after time points when significant proficiency changes occurred could be useful in improving the EBUS-TBNA learning environment.
The findings of this pilot study indicate that CUSUM analysis was useful in detecting deviations in bronchoscopist performance and identifying environmental factors associated with such performance. This is the first study to show that CUSUM analysis could be used to investigate the EBUS-TBNA learning environment and identify possible causes for performance declines. We believe that CUSUM analysis can hasten intervention when changes in performance levels are detected.
Both individual and institutional factors affect successful performance of EBUS-TBNA. In this study, we found that presence of additional assisting attending bronchoscopists was associated with successful performance of EBUS-TBNA, which suggests that having an assisting attending bronchoscopist was associated with successful procedures during the institutional learning phase at our center. During the study period, additional attending physicians assisted in identifying target lymph nodes and needle activation. When an assisting attending physician was not present, a respiratory therapist or fellow physician assisted with the procedures. It is possible that the presence of an assisting attending physician resulted in better identification of target lymph nodes, better sampling technique, or both. Operator B exhibited the steepest learning curve, while operator A had the slowest learning curve among the operators. Further quality improvement efforts could attempt to use the present results for detailed investigation and comparison of the learning environments of these 2 operators. For example, operator B might have had greater input from other bronchoscopists. Interestingly, the presence of cytopathologists for rapid on-site evaluation in the bronchoscopy suite did not seem to affect the likelihood of a successful procedure. This finding contradicted our expectations, as we had assumed that the availability of a cytopathologist for rapid on-site evaluation would positively affect performance because lymph node sampling could then be repeated until satisfactory samples were obtained. The change in proficiency during the worsening interval did not appear to coincide with the change in the ratio of procedures performed by the bronchoscopist.
We used new definitions of EBUS-TBNA success and failure. We believe that these definitions are better for determining bronchoscopic proficiency than those used by Kemp et al. , as our definitions do not penalize bronchoscopists for the inherent limitations of the FNA procedure.
Study limitations and implications
This study has several limitations, which are mainly the result of its retrospective design. We were able to obtain information on only a limited number of environmental factors in our bronchoscopy suite. In particular, we lacked data on the technique and performance-based factors. We also had no information on changes in bronchoscopist lifestyle that might have affected performance. Additionally, we had no way of determining the “degree of difficulty” for each attempted sample. The presence of subcentimeter lymph nodes in less accessible locations might decrease success rate, even for a moderately proficient bronchoscopist. To address these limitations and increase the feasibility of this method for quality improvement, our next investigation will be a prospective study. The variables assessed will be more comprehensive, including bronchoscopist work hours, sleep hours, technique and performance-based factors, teaching methods, location and size of lymph nodes, and factors involved in specimen processing.
Combined CUSUM and environmental analysis is a promising method for monitoring and improving performance in EBUS-TBNA.
Endobronchial ultrasound-guided transbronchial needle aspiration
The authors wish to thank David Kipler for reviewing the language of the manuscript.
This study received no external funding.
Availability of data and materials
The dataset supporting the conclusions of this article is presented within the article. The detailed clinical dataset is not publicly available, to protect research subject privacy and confidentiality.
YN and MJ designed the study. DS supervised the project. MJ collected the data. YT, RU and SF analyzed the data. YN, RU and SF wrote the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
The Ethical Committee of Saint Louis University approved the study and waived the need for informed consent in 2011.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Page E. Continuous inspection schemes. Biometrika. 1945;41:100–15.View ArticleGoogle Scholar
- Bolsin S, Colson M. The use of the Cusum technique in the assessment of trainee competence in new procedures. Int J Qual Health Care. 2000;12(5):433–8.View ArticlePubMedGoogle Scholar
- Hurter T, Hanrath P. Endobronchial sonography: feasibility and preliminary results. Thorax. 1992;47(7):565–7.View ArticlePubMedPubMed CentralGoogle Scholar
- Dhooria S, Gupta N, Bal A, Sehgal IS, Aggarwal AN, Sethi S, et al. Role of Xpert MTB/RIF in differentiating tuberculosis from sarcoidosis in patients with mediastinal lymphadenopathy undergoing EBUS-TBNA: a study of 147 patients. Sarcoidosis Vasc Diffuse Lung Dis. 2016;33(3):258–66.PubMedGoogle Scholar
- Kemp SV, El Batrawy SH, Harrison RN, Skwarski K, Munavvar M, Rosell A, et al. Learning curves for endobronchial ultrasound using cusum analysis. Thorax. 2010;65(6):534–8.View ArticlePubMedGoogle Scholar
- Kestin IG. A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth. 1995;75(6):805–9.View ArticlePubMedGoogle Scholar
- Sivaprakasam J, Purva M. CUSUM analysis to assess competence: what failure rate is acceptable? Clin Teach. 2010;7(4):257–61.View ArticlePubMedGoogle Scholar