Validation of the Pneumocystis pneumonia score in haematology patients with acute respiratory failure

Background Pneumocystis pneumonia (PCP) is an important cause of acute respiratory failure (ARF) in immunocompromised patients, yet no actual clinical tool suitably identifies patients at risk. Recently, a multivariable prediction model has been proposed for haematology patients with ARF requiring intensive care unit (ICU) admission to assess the risk of PCP (PCP score). However, it has not yet been validated externally. Methods To validate the PCP score, a retrospective cohort study was conducted in two large designated haematology centres in Korea. One-hundred and forty haematology patients with ARF were admitted to ICU. They underwent aetiologic evaluations between July 2016 and June 2019. The predictive ability of the score was assessed with the receiver operating characteristic (ROC) curve analysis for both the discrimination and calibration of the score. Results Among the 141 patients, 13 (9.2%) were finally diagnosed of PCP. Although the median of PCP score in PCP group was higher than in non-PCP group (3.0 [interquartile range 0.0–4.0] vs. 2.0 [0.5–4.0]), the difference was not statistically significant (P = 0.679). The area under the ROC curve of the PCP score in our cohort was 0.535 (95% CI, 0.449–0.620), indicating no discriminatory ability. When using a cut-off of 3.0 the score, the result was 38.5% (95% CI, 13.9–68.4) sensitive and 7.03% (95% CI, 61.6–78.1) specific. The negative predictive value was 58.8% and positive predictive value was 59.8% for a 10% prevalence of PCP. Conclusions In this study, the PCP score was not useful to predict the risk of PCP in haematology patients with ARF. Further prospective validation studies are needed to validate the score’s use in routine clinical practice for the early diagnosis of PCP in haematology patients.


Background
Pneumocystis pneumonia (PCP), a pulmonary infection caused by Pneumocystis jirovecii, is an important cause of acute respiratory failure (ARF) in immunocompromised patients [1][2][3][4][5]. PCP is most commonly associated with the human immunodeficiency virus (HIV) infection. However, the development of highly active antiretroviral therapy and effective prophylaxis against PCP have reduced its prevalence and the mortality rate in HIV-positive patients [6,7]. Consequently, more attention is placed on other immunocompromised states [8][9][10].
Haematologic malignancies are the most common underlying conditions associated with the development of PCP in HIV-negative patients [11,12]. Compared to HIV-positive patients who follow a more indolent course [13,14], haematology patients with PCP present with abrupt-onset hypoxemic respiratory failure, and more often require mechanical ventilation [13][14][15]. In addition, delays in anti-PCP treatment are associated with poor outcomes [13,16]. However, inappropriate use of TMP/SMX should be avoided not only prevent drug related side effects including granulocytopenia, hepatotoxicity, and nephrotoxicity, but also prevent delay of inaccurate diagnosis or resistant strains. Nonetheless, the confirmative diagnosis of PCP with bronchoalveolar lavage (BAL) would be challenging in haematology patients with hypoxemic respiratory failure, making it difficult to perform bronchoscopy [10,14]. Therefore, a clinical tool that rapidly identifies patients at risk of PCP (in whom empiric treatment is warranted), should be developed and consequently avoiding delays in anti-PCP treatment [17].
Recently, Azoulay et al. introduced a multivariable prediction model to assess the risk of PCP (PCP score) for haematology patients with ARF requiring ICU admission and they reported a good performance of the PCP score [18]. However, there have been no external validations of the prediction model with other cohorts. In this study, we then assessed the performance of the PCP score in haematology patients from two large designated haematology centres in Korea.

Methods
We retrospectively reviewed the medical records of all consecutive haematology patients admitted in the medical intensive care unit (ICU) for respiratory failure at Samsung Medical Center (a 1989-bed, university-affiliated, tertiary referral hospital in Seoul, South Korea) and Seoul St. Mary's Hospital (a 1369bed, university-affiliated, tertiary referral hospital in Seoul, South Korea) between July 2016 and June 2019. The Institutional Review Boards of each participating hospital with patient records approved the present study and the informed consent was waived because of the non-interventional nature of this research. All patient records and data were anonymised and coded prior to analysis.

Study population
All consecutive haematology patients older than 20 years who were admitted to the medical ICU for ARF were screened for inclusion. Patients were included if they received BAL with or without a transbronchial lung biopsy (TBLB) for aetiologic explorations, and a results of microbiological identification of Pneumocystis jirovecii in BAL fluid or lung tissue were included (Fig. 1). Patients were excluded if they had a positive HIV antibody test. For cases with multiple admissions for ARF during the study period, only the first ICU admission was evaluated.
Pneumocystis pneumonia score Azoulay et al. developed a PCP score for haematology patients with ARF using a cohort with PCP prevalence of 11.2% (149 of 1330), in which PCP was confirmed by identification of P. jirovecii cysts or trophozoites in BAL fluid or induced sputum [18]. The PCP score uses parameters: age, lymphoproliferative disease, anti-PCP prophylaxis, day between respiratory symptom onset and ICU admission, shock at ICU admission, and chest X-ray results (showing pleural effusion or not, gotten at ICU admission). The PCP score ranged from -6 to 8.5, and higher scores indicated more possibility of PCP and the cut-off value of the PCP score was found to be 3.0 from the validation cohort.

Data collection
Data extracted from the medical records include general demographic information, underlying haematologic disease and medications received during the previous month. Furthermore, PCP prophylaxis, initial presentation of symptoms, vital signs/organ supports at ICU admission and chest radiographic findings. We also collected laboratory data such the white blood cell count, albumin, C-reactive protein, procalcitonin, lactate dehydrogenase, and haemoglobin. Severity of illness was assessed using the Sequential Organ Failure Assessment (SOFA) score [19]. The data included integral components of the PCP score [18].
The diagnosis of PCP was based on the identification of the organism in BAL fluid or lung tissue obtained by TBLB. BAL fluid samples were stained using Gram and Ziehl-Neelsen methods and then cultured for bacteria, mycobacteria and fungi. Microbiological identification of P. jirovecii was confirmed by documenting the organism with Wright Giemsa or Gram-Weigert stain or the cyst with Gomori methenamine silver or calcofluor white stain [1], which is the same methods with the previous study [18].

Statistical analysis
Data was presented as medians and interquartile ranges (IQR) for continuous variables and as numbers and percentages for categorical variables. The baseline and clinical characteristics on ICU admission were compared between the PCP group and the non-PCP group using the Mann-Whitney U test (for continuous variables) and the Pearson's chi-square test or Fisher exact test (for categorical variables).
To determine the sample size needed for the validation of our prediction model [20], we based our calculations on the test characteristics of the PCP score. For a ROC curve area of 0.8, a power (type II error) of 80% and an α (type I error) of 0.05, the sample size required for our study was 103 (8 from the positive group and 95 from the negative group). This was calculated using MedCalc statistical software.
The predictive ability of PCP score was assessed with ROC curve analysis for both the discrimination (via the C index) and calibration (using Hosmer-Lemeshow statistics) of the score [21]. Univariable logistic regression analysis was performed to estimate the odds ratios (ORs) of each variable in the PCP score. The ORs of each variable were reported with their 95% confidence intervals (CIs). Sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively) were calculated for the PCP score. All variables were analysed using R Statistical Software (Version 3.2.5; R Foundation for Statistical Computing, Vienna, Austria).

Performance of the PCP score
The area under the ROC curve was 0.535 (95% CI, 0.449-0.620), indicating no discriminatory ability in our haematology patients with ARF (Table 3, Fig. 2). Comparisons of PCP scores between PCP and non-PCP groups are illustrated in Fig. 3, but no significant difference was observed (P = 0.679). Of 13 patients, 7 (53.8%) received a PCP score of 3.0 or higher. When using a cut-off PCP score value of 3.0, a sensitivity of 38.5% (95% CI, 13.9-68.4) and a specificity of 7.03% (95% CI, 61.6-78.1) were obtained. The negative predictive value was 58.8% and the positive predictive value was 59.8% for a 10% prevalence of PCP. Performances of other cut-off values are reported in Table 4.
All variables of the PCP score were compared between the PCP and the non-PCP groups (Table 5). There was no significant difference between the two groups. In addition, the results of univariable analysis (using logistic regressiion model) of variables in the PCP score also showed no significant difference between the two groups.

Discussion
Over the last decade, there has been a substantial decline in PCP-related mortality rates among HIV-positive patients. Conversely, there is an increasing mortality rates associated with PCP in HIV-negative patients [14,15]. In addition, a delay in anti-PCP treatment in these patients is associated with a higher mortality rate [13,16]. This finding suggests that empiric therapy for PCP should be initiated in patients with high clinical suspicion for PCP. Unfortunately, there is no clinical tool that rapidly identifies patients at risk of PCP. Therefore, a high index of suspicion using patient history and clinical presentation, are key factors in early diagnosis of PCP [22]. However, the clinical picture varies individually as the general hallmarks (fever, shortness of breath and diffuse infiltrates) of this disease do not consistently occur, especially in its early course [23]. Therefore, clinical diagnosis is complicated because no combination of symptoms, signs, blood chemistries and/or radiographic findings is specific for PCP [24].
Recently, Azoulay et al. suggested a multivariable predictive model to improve the early diagnosis of PCP in haematology patients with ARF requiring ICU admission [18]. Variables included in the model were age, lymphoproliferative disease, anti-PCP prophylaxis, number of days between onset of respiratory symptoms and ICU admission, shock, chest radiograph pattern, and pleural effusion. Higher scores, were associated with lymphoproliferative disease, no anti-PCP prophylaxis, more than a three-day duration between onset of respiratory symptoms and ICU admission, and no alveolar pattern on radiography. Meanwhile lower scores were associated with those However, there are several questions regarding the variables used in the final model. Firstly, those patients over the age of 50 years were associated with lower scores meaning a lower risk of PCP. The authors described these points to be in line with older patients receiving less frequently high-dose chemotherapy or stem cell transplantation. These conditions put patients at high risk for PCP. However, the majority of haematologic malignancies are diagnosed in elderly patients and  Values are given as median (interquartile range) or number (percentage) ICU intensive care unit, SOFA sequential organ failure assessment, WBC white blood cell, ALC absolute lymphocyte count, PCP Pneumocystis pneumonia the decision to treat might not only be determined by age, but also by combining performance and frailty [25,26]. In addition, previous reports showed an association between age > 60 years and pulmonary Pneumocystis colonisation, especially in patients with rheumatoid arthritis [27,28]. Secondly, chest X-ray findings of PCP are non-specific and sometimes normal. In some cases, PCP presents bilateral, symmetric opacities in an interstitial or alveolar pattern on a chest X-ray. This is associated with an increased frequency of spontaneous pneumothorax [23,29,30]. Therefore, high resolution computed tomography (known to be the most reliable imaging technique for the detection and differential diagnosis of PCP), is recommended in immunocompromised patients, complementing either a negative or a vague chest Xray [31].
Ideally, critically ill patients should be admitted in the ICU as soon as possible to receive the best appropriate care. However, delays in admission are common due to triage, diagnostic and logistic reasons [32][33][34]. Therefore, the duration between respiratory symptom onset and ICU admissions vary depending on each hospital's policy and ICU bed availability. Our study also showed delayed ICU admissions (5 days versus 10 days in the PCP group). In addition, PCP patients presented more shock (22.4% versus 38.5% in the PCP group) and pleural effusion (5.2% versus 23.1%) in our cohort which are inconsistent with the PCP scores in Azoulay et al. These results are explained by the delayed ICU admission in our cohort.
To the best of our knowledge, this is the first external validation of the PCP score, with varying predictive results in haematology patients from the Korean cohort. However, there were several potential limitations to our study. Firstly, although the sample size was larger than a priori (which was adequately powered to obtain the significant of an area under ROC curve of 0.8), it was relatively small compared to the  original study. However, post hoc power analysis revealed that the observed power was 96.4% with a 9.2% prevalence of PCP in our cohort. Thus, the results are adequately powered to rule out the possibility of false-negative findings. Secondly, the diagnosis of PCP was confirmed by the identification of the organism in BAL fluid or lung tissue only. The diagnostic strategy for PCP now usually combines noninvasive diagnostic tests and PCR testing of BAL fluid. However, quantitative PCR was not yet available in Korea. Therefore, patients with positive PCR related to colonisation were considered as not having PCP in this study.

Conclusion
From the analysis of the two cohorts from designated haematology centres in Korea, the PCP score were not useful to predict the risk of PCP in haematology patients. Further prospective studies are needed before the score can be implemented into routine clinical practice for the early diagnosis of PCP in haematology patients.