Increasing the accuracy of the asthma diagnosis using an operational definition for asthma and a machine learning method

Abstract

Introduction

Analysis of the National Health Insurance data has been actively carried out for the purpose of academic research and establishing scientific evidences for health care service policy in asthma. However, there has been a limitation for the accuracy of the data extracted through conventional operational definition. In this study, we verified the accuracy of conventional operational definition of asthma, by applying it to a real hospital setting. And by using a machine learning technique, we established an appropriate operational definition that predicts asthma more accurately.

Methods

We extracted asthma patients using the conventional operational definition of asthma at Seoul St. Mary’s hospital and St. Paul’s hospital at the Catholic University of Korea between January 2017 and January 2018. Among these extracted patients of asthma, 10% of patients were randomly sampled. We verified the accuracy of the conventional operational definition for asthma by matching actual diagnosis through medical chart review. And then we operated machine learning approaches to predict asthma more accurately.

Results

A total of 4,235 patients with asthma were identified using a conventional asthma definition during the study period. Of these, 353 patients were collected. The patients of asthma were 56% of study population, 44% of patients were not asthma. The use of machine learning techniques improved the overall accuracy. The XGBoost prediction model for asthma diagnosis showed an accuracy of 87.1%, an AUC of 93.0%, sensitivity of 82.5%, and specificity of 97.9%. Major explanatory variable were ICS/LABA,LAMA and LTRA for proper diagnosis of asthma.

Conclusions

The conventional operational definition of asthma has limitation to extract true asthma patients in real world. Therefore, it is necessary to establish an accurate standardized operational definition of asthma. In this study, machine learning approach could be a good option for building a relevant operational definition in research using claims data.

Introduction

Claims data-based studies have become common in health care research during the past decade [1]. Claims data are attractive to researchers because they offer numerous advantages. Claims data have health information that is anonymous, abundant, inexpensive, and widely available in an electronic format [1], and they reflect real-world medical practice. Therefore, these data have been utilized for academic research, post-market surveys, and economic evaluations. However, claims data have several disadvantages; they are not designed for medical research, as they are occasionally missing critical values, and they are under or over-reported in real clinical data [2]. In addition, disease codes in the claims data may not represent a patient’s true disease status. Therefore, building an accurate operational definition of claims data is very important to make these studies more relevant.

Several studies have used national claims data in asthma research [3,4,5,6,7]. In most of these studies, the researchers extracted the asthma patients using a conventional operational definition, which contains two major components. First, they must have the International Classification of Diseases Tenth Revision (ICD-10) codes of asthma as the major diagnosis. Second, they must have been prescribed asthma-related medication. However, there are concerns in asthma research as to whether asthma patients extracted through a conventional operational definition, reflect the reality of all clinical situations.

In this study, we verified the accuracy of the conventional operational definition of asthma by applying it to a real hospital setting. We established an appropriate operational definition that predicts asthma more accurately, using a machine learning technique.

Methods

Study design and population

We extracted asthma patients using the conventional operational definition of asthma at Seoul St. Mary’s Hospital (1,356-bed tertiary referral hospital) and St. Paul’s Hospital (301 beds) at the Catholic University of Korea between January 2017 and January 2018. The conventional operational definition of asthma was: 1) ICD-10 codes for asthma; 2) use of more than one drug as an asthma-related medication on at least two outpatient clinic visits [inhaled corticosteroids (ICSs), long-acting β2-agonists (LABAs), ICSs plus long-acting β2-agonists (ICS/LABAs), long-acting muscarinic antagonists (LAMAs), short-acting β2-agonists, short-acting muscarinic antagonists, leukotriene receptor antagonists, systemic β 2-agonists, systemic corticosteroids, or xanthine derivatives] [8].

About 10% of these extracted asthma patients were randomly sampled. We excluded patients < 19-years and patients who did not visit the pulmonology or allergy clinic during the study. We verified the accuracy of the conventional operational definition for asthma by matching the actual diagnosis in a medical chart review. Then, we operated a machine learning approach to predict asthma more accurately. All methods were performed in accordance with the relevant guidelines and regulations. The present study was approved by the Institutional Review Board of Seoul St. Mary’s Hospital and was exempted from informed consent requirements owing to its retrospective design (KC18ZNSI0850).

Predictors

The predictors for model development were chosen from the results of pulmonary function tests and asthma-related medications in the conventional operational definition of asthma.

Statistical analysis

We developed a reference model and five machine learning models to predict asthma in the training set (randomly selected 75% of the sample). We fit a logistic regression model for the reference model, including all of the predictors. The predictive models were built with: (1) a decision tree, (2) random forest, (3) extreme gradient boosting (XGBoost), (4) light gradient boosting machine (LGBM), and (5) the CatBoost algorithm. Hyperparameter optimization was performed by a grid search and Bayesian optimization.

A decision tree is a non-parametric supervised learning method used for classification and regression. It is a flowchart-like tree structure, where the internal nodes denote a test of an attribute; each branch represents an outcome of the test, and each leaf node holds a class label. Random forest is an ensemble of decision trees created using bootstrap samples of the training data and random feature selection for inducing the tree. The LGBM was designed to be accurate, efficient, and quick, which are advantages when handling large-scale data. XGBoost is an implementation of a gradient boosting machine and one of the best-performing algorithms utilized for supervised learning. It can be used for both regression and classification problems. CatBoost provides a gradient boosting framework that attempts to solve for categorical features using a permutation-driven alternative compared to the classical algorithm.

We measured the predictive performance of each model by computing the area under the receiver operating characteristic curve (AUC), as well as the accuracy, sensitivity, and specificity of the F1-measure to assess model quality in the test set (remaining 25% of the sample). In addition, to gain stable predictions, we measured the predictive performance of each model with tenfold cross-validation. To test the difference in the evaluation index of each model, this study used bootstrapping on the extra-validation data, prepared 1,000 different test sets in 50 unit sizes or 50,000 bootstrap samples, and applied analysis of variance to test the difference in the average value of the index. Tukey’s post-hoc test was used. All analyses were performed with R version 3.4 software (The R Foundation for Statistical Computing, Vienna, Austria).

Results

Baseline characteristics of the study population

A total of 4,235 patients with asthma were identified using the conventional asthma definition. Of these, 353 patients were enrolled (Fig. 1). The baseline characteristics of the enrolled patients are described in Table 1. Among the 353 patients, 49.3% were female and the mean age was 64.6 years; 45.3% were never smokers, 34.8% were current or ex-smokers, and the average smoking pack-years was 18.3. The mean post-bronchodilator (BD) forced expiratory volume in 1 s (FEV1) was 2.01 L (76.3% of predicted) and the mean post-BD FEV1/forced vital capacity was 65.6. Table 2 shows the maintenance drugs of the study population. An ICS/LABA combined inhaler was the most commonly prescribed medication, followed by leukotriene receptor antagonist (LTRA) and LAMA inhalers.

Fig. 1

Flow diagram for patient enrollment

Table 1 Baseline characteristics of the study population

Table 2 Maintenance drugs taken by the study population

Actual diagnosis by medical chart review

Figure 2 shows the actual diagnoses of the study population. The patients with asthma comprised 56% of the study population, and 44% of the patients did not have asthma. Of these, chronic obstructive pulmonary disease (COPD) was the most common, followed by bronchitis, bronchiolitis obliterans (BO), and other diseases. Additionally, we prescribed asthma medications according to the actual diagnosis (Fig. 3). A LAMA inhaler was the most frequently prescribed medication for COPD and BO patients.

Fig. 2

Actual diagnoses of the study population

Fig. 3

Prescribed medications according to the actual diagnosis

Predicting asthma patients using a machine learning technique

Two cross-validation methods were used. The predictive abilities of the machine learning models in the asthma diagnosis are summarized in Tables 3 and 4. The LGBM prediction model (grid search) for the asthma diagnosis had an accuracy of 85.9%, an AUC of 91.7%, sensitivity of 79.0%, and specificity of 100%. These results were similar to those of logistic regression and better than those of random forest. Figure 4 shows the decision tree model results (after a grid search) for the asthma diagnosis. The accuracy of the decision tree model was 81.2%. Accuracy was higher than actual accuracy, although accuracy was lower than the other models. The XGBoost predictive model (Bayesian hyperparameter tuning) for the asthma diagnosis showed an accuracy of 87.1%, an AUC of 93.0%, sensitivity of 82.5%, and specificity of 97.9%. This result was better than that of logistic regression. The other models showed better results than actual accuracy. The machine learning models provided higher predictive ability than the conventional operational definition of asthma.

Table 3 Predictive abilities of the machine learning models for the asthma diagnosis after a grid search with cross-validation

Table 4 Predictive abilities of the machine learning models for the asthma diagnosis after cross-validation with Bayesian hyperparameter tuning

Fig. 4

Decision tree model for the asthma diagnosis. Entropy is a measure of the randomness in the information being processed. The higher the entropy, the harder it is to conclude. The sample is the number of data corresponding to the current node. The value list indicates how many samples from a given node fall into each category. Each element in the classification domain is called a class

Accuracy was significantly different as a result of bootstrapping, compared with XBG (Bayes), which showed the highest accuracy, but the difference in the AUC between the three models was not significant. The three models with the highest performance had a similar degree of accuracy (Table 5).

Table 5 Predictive abilities of the machine learning models for the asthma diagnosis after bootstrap cross-validation

Important predictive variables

Figures 5 and 6 indicate the important variables for predicting the asthma diagnosis. The major explanatory variables were ICS/LABA, LAMA, and LTRA for a proper diagnosis of asthma. The most significant variable for a proper diagnosis of asthma in the logistic regression model was using an ICS/LABA inhaler, followed by an ICS inhaler (Table 6). This result is similar to the result of the machine learning technique.

Fig. 5

Variable importance based on the machine learning models for the asthma diagnosis after a grid search with cross-validation

Fig. 6

Variable importance based on the machine learning models for the asthma diagnosis after cross-validation with Bayesian hyperparameter tuning

Table 6 Variable importance using logistic regression to predict the asthma diagnosis more accurately

Discussion

Claims data can be used to examine the current status and trends that reflect the actual health care environment rather than a limited experimental environment. However, the accuracy of the disease diagnosis included in claims data remains controversial. An operational definition can be used to confirm the presence or absence of disease. However, this approach has some limitations. The number of extracted patients decreases, which dilutes the advantage of claims data as big data when placing conditions for extracting patients using the operational definition. Although extracting with simple conditions leads to extracting more patients, it results in including more unintended patients.

In this study, we examined the actual proportion of patients by comparing the patients sampled based on an operational definition and the data in their medical charts. The data were analyzed based on an operational definition frequently used in previous research. Patients with asthma accounted for only 56% of the cases. In other words, 44% were erroneously included as asthma patients even if they did not have asthma. To overcome this problem, we used a machine learning approach that resulted in an observable increase in accuracy. To the best of our knowledge, this is the first study to compare claims data to those in actual medical charts. Additionally, this is the first study to utilize a machine learning method to increase the accuracy of the diagnosis of patients sampled via an operational definition.

The key predictor in the machine learning models was the ICS/LABA inhaler. ICS/LABA are basic drugs for treating asthma. Therefore, many asthmatic patients use ICS/LABA. ICS/LABA are often used for COPD, and ICS/LABA are still in use for some airway diseases. Thus, determining whether someone has asthma with ICS/LABA is less accurate. Another key predictor was LAMA. The use of LAMA was identified by logistic regression as least associated with the diagnosis of asthma. LAMA is the most basic treatment for COPD but is used as an add-on in severe asthma cases. Previous studies that used claims data were inconsistent in including LAMA as an asthma-related drug. Nevertheless, the sample sizes of two other studies conducted during the same time were different due to a difference in the operational definition. The proportion of LAMA use is 20% in tertiary hospitals [7]. LAMA use has been excluded in studies that did not include LAMA [3]. In our study, a review of the medical charts of the patients in the sample showed that COPD was the second most common wrong entry after asthma and that some COPD patients used LAMA. However, if LAMA was removed, 13.1% (26/199) of actual asthma patients would be excluded. In particular, severe asthma patients would be excluded. However, if a machine learning method is used, the accuracy of the asthma diagnosis increases, even when patients are sampled via an operational definition that included LAMA.

According to our study, the accuracy of previous definition of asthma by claim data was only 56%. Thus, many previous studies by conventional definition may not represent real asthma patients. This is one of the reasons why we performed this study. By applying this new method, researchers can analyze more accurate characteristics of asthma patients in the future.

Our study had several limitations. First, the patients were older. The prevalence of asthma increases with age; however, the patients in this study were much older than those included in previous studies that used claims data [9, 10]. The higher the age of an asthma patient, the more difficult it is to differentiate asthma from COPD [11,12,13,14]. However, even when considering these factors, accuracy increased in our study. Second, only the patients in a referral hospital were extracted and analyzed. There are differences in the use of medications in primary, secondary, and tertiary hospitals [7]. The frequency of use of LAMA in a primary hospital is remarkably low. The key predictor in the machine learning methods was LAMA. Thus, the effectiveness of machine learning may seem exaggerated, but the increase in diagnostic accuracy is undeniable. Third, our study used the multiple machine learning models and hyper-parameter optimization techniques may increase the risk of overfitting the models to the training data, which may result in poorer performance when applied to new data. We will keep this in mind when applying these techniques to other studies. Fourth, the sample size was small. Because it took a long time to review the medical charts of all patients in the sample, only 10% of the medical charts were reviewed randomly. However, this sample was sufficiently large to apply machine learning.

Conclusion

The conventional operational definition of asthma has a limited range, so that true asthma patients in the real world may be extracted. Therefore, it was necessary to establish an accurate standardized operational definition of asthma. In this study, a machine learning approach was a good option for building a relevant operational definition using claims data.