Skip to main content

Verifying a questionnaire diagnosis of asthma in children using health claims data

Abstract

Background

Childhood asthma prevalence is widely measured by parental proxy report of physician-diagnosed asthma in questionnaires. Our objective was to validate this measure in a North American population.

Methods

The 2884 study participants were a subsample of 5619 school children aged 5 to 9 years from 231 schools participating in the Toronto Child Health Evaluation Questionnaire study in 2006. We compared agreement between "questionnaire diagnosis" and a previously validated "health claims data diagnosis". Sensitivity, specificity and kappa were calculated for the questionnaire diagnosis using the health claims diagnosis as the reference standard.

Results

Prevalence of asthma was 15.7% by questionnaire and 21.4% by health claims data. Questionnaire diagnosis was insensitive (59.0%) but specific (95.9%) for asthma. When children with asthma-related symptoms were excluded, the sensitivity increased (83.6%), and specificity remained high (93.6%).

Conclusions

Our results show that parental report of asthma by questionnaire has low sensitivity but high specificity as an asthma prevalence measure. In addition, children with "asthma-related symptoms" may represent a large fraction of under-diagnosed asthma and they should be excluded from the inception cohort for risk factor studies.

Peer Review reports

Background

Parental proxy report of physician-diagnosed asthma (questionnaire diagnosis) is the standard measure of childhood asthma prevalence used in national health surveys and epidemiological studies. The International Study of Allergies and Asthma in Childhood (ISAAC) questionnaire is the current gold standard for ascertaining asthma outcomes in epidemiologic studies [1]. The "ever wheeze" question is often used to compare asthma prevalence between countries. For international studies, a symptom-based definition is less subject to bias than a diagnosis-based definition[2]. However, for national studies that evaluate risk factors, a more specific definition is often desired [3–5]. In these studies, lifetime asthma is measured by affirmative response to the question: "Has your child ever had asthma?" and is further defined to be "Doctor diagnosed asthma" if there is an affirmative response to the question "Has this been diagnosed by a doctor?" Despite its widespread use in studies assessing asthma prevalence[2], risk factors [6],and diagnostic tests [7] this questionnaire based asthma diagnosis has not been validated in a North American population.

Canada has a universal health care system in which all Canadians have equal access to physician and hospital services. Health claims data on all patient encounters is collected for administrative purposes although in Ontario there is currently no centralized database on prescription medication. These databases have been found to be accurate compared to chart abstraction for diseases such as ischemic heart disease[8], esophagitis[9] as well as asthma[10]. Recently, an algorithm has been developed to identify children with asthma from health claims data in the province of Ontario. This algorithm, has been shown to have 91.4% sensitivity and 82.9% specificity for correctly identifying asthma when compared to expert consensus diagnosis of asthma[10, 11].

The objective of this research was to assess agreement between questionnaire based parental proxy report of physician diagnosed asthma in children (hereafter referred to as "questionnaire diagnosis") and asthma diagnosed by analyzing health claims data (hereafter referred to as "health claims diagnosis") in a population-based sample of urban Canadian school children.

Methods

Subjects

Participants were recruited from The Toronto Child Health Evaluation Questionnaire (T-CHEQ) study. This study used ISAAC methodology[1] to recruit a population based sample of 5619 children (aged 5 to 9 years) from 231 Toronto public schools between January and May of 2006. The demographic characteristics of these children closely resembled census data[12] and the prevalence of asthma outcomes closely resembled national health survey data[12]. Detailed methods for sampling and recruitment are published elsewhere[12]. All subjects that participated in the T-CHEQ study were asked at the time of the initial study if they would consent to have their child's questionnaire linked to health claims data for research purposes and those that agreed were included in this study.

Study Design

This is a diagnostic validation study, comparing agreement between questionnaire asthma diagnosis (using data from the cross-sectional T-CHEQ study) and health claims diagnosis (using cohort data from lifetime health claims administrative databases).

Questionnaire based Asthma diagnosis

Asthma diagnosis was identified from the T-CHEQ study sample by affirmative responses to the questions "Has your child ever had asthma?" and "Was this diagnosed by a doctor?" Those that reported non-physician diagnosed asthma were excluded from the analysis. Non-asthma controls did not report doctor diagnosed asthma. These controls were further categorized into those with "Asthma-related symptoms" if they reported a yes response to either "Has your child ever had wheezing" or "In the past 12 months has your child had a dry cough at night, apart from a cough associated with a cold or chest infection?".

Health Claims Asthma Diagnosis (Reference Standard)

Health claims data between March 31, 1997 and March 31, 2006 (i.e. the child's lifetime) from two Ontario health care administrative databases were used: (1) the Canadian Institute for Health Information (CIHI) discharge abstract database for inpatient services and (2) The Ontario Health Insurance Plan (OHIP) for ambulatory and emergency services. Both of these databases contain diagnostic codes based on the International Classification of Disease (ICD)-9 or 10. A claim for asthma was identified by the ICD- 9 code 493 for claims up to March 31, 2002 and ICD-10 codes J45 and J46 identified asthma in subsequent years. The CIHI database currently allows up to 25 diagnostic codes (prior to 2002, 16 diagnostic codes were allowed) and if any of these was for asthma, the hospitalization was included. The OHIP database allows one diagnostic code per visit. Only one claim per physician per day per patient was allowed. The health claims databases were also linked to the Registered Persons Database which contains mortality and demographic data to ensure that the subjects had lived within the province since birth. A unique personal identifier (the scrambled Health Card Number) included in each database permits the linkage of a child's records across all databases and time while preserving patient confidentiality.

Prevalent asthma cases were defined by a previously validated algorithm as follows: at least one hospitalization for asthma at any time during the child's life or two separate ambulatory or emergency room visits for asthma within a two year time frame[11]. This algorithm was previously found to have optimal diagnostic parameters using an expert consensus diagnosis from chart abstraction[10, 11].

Data linkage

The T-CHEQ participants were anonymously linked to the health claims databases through their reported Health Insurance Number. A matching date of birth in the two databases was also needed for the link to be considered valid. All data linkage and analysis related to this study was completed within the secure confines of the Institute of Clinical and Evaluative Sciences in Toronto, Ontario.

Statistical methods

"Questionnaire diagnosis" and "health claims diagnosis" were compared in two by two tables. Sensitivity and specificity of the questionnaire diagnosis were calculated using health claims diagnosis as the gold standard. In order to test the potential misclassification bias that could occur by including children with asthma-related symptoms in the non-asthma control group, the children with asthma-related symptoms were removed from the sample and sensitivities and specificities were recalculated. Additional sensitivity analyses were conducted by modifying the health claims data algorithm (increasing the time frame for incident asthma ambulatory claims from 2 to 3 years or including emergency visit data from an additional database). We also calculated agreement (kappa)[13] between questionnaire and health claims diagnosis.

Ethical Approval

Parents of children in this study gave informed consent for participation in this research by filling in the voluntary T-CHEQ questionnaire and additionally agreeing to participate in the data linkage. This study was approved by the Research Ethics Board of the Hospital for Sick Children in Toronto.

Results

Baseline characteristics

From the original TCHEQ cohort, 2884 (51.32%) gave permission to link to health claims data. Respondents did not differ from non-respondents in terms of asthma prevalence or gender, however they were more likely to be in a higher income group (42.13% versus 31.40%) and have post-graduate education (51.96% versus 40.09%), and were less likely to report no physician visits in the past year (24.85% versus 29.57%) (Table 1). Data linkage was successfully achieved in 2782 of the respondents (96.46%).

Table 1 Characteristics of Toronto School Children with Asthma Questionnaire and Health Claims Data, 2006a

Asthma prevalence

In our study sample, 437 children were identified with questionnaire diagnosis of asthma (prevalence of 15.71%) while 586 children were identified with health claims diagnosis of asthma (prevalence of 21.06%) (Table 2). The vast majority of subjects with ever asthma (defined by the question "Has your child ever had asthma?") had physician diagnosed asthma (also responded affirmatively to the question "Was this diagnosed by a doctor?") (prevalence of asthma 16.41% and 15.71% respectively).

Table 2 Questionnairea Asthma Diagnosis versus Health Claimsb Diagnosis (n = 2782)

Questionnaire accuracy

The questionnaire diagnosis had a sensitivity of 59.04% and a specificity of 95.86% for detecting asthma using the health claims diagnosis as the reference standard.

Of the 2435 non-asthma controls, 854 (35%) had asthma-related symptoms. When these children were removed from the sample (table 3), the sensitivity and specificity of the questionnaire diagnosis were 83.57% and 93.56% respectively.

Table 3 Questionnairea Asthma Diagnosis versus Health Claimsb Diagnosis, excluding children with "asthma-related symptoms" (n = 1826)

The sensitivity analysis performed using modified algorithms for definitions of health claims diagnosis did not produce any significantly different findings (data not shown).

We observed moderate agreement (kappa = 0.60) between questionnaire asthma definitions and health claims asthma definitions. Good agreement (kappa = 0.75) was observed when those with asthma-related symptoms were excluded.

Discussion

Our findings concur with other literature that suggests that questionnaire asthma diagnosis is specific but not sensitive for asthma[14–17]. As expected, compared with questionnaires that use a definition of "wheezing in the last twelve months" to define the population with asthma, our definition of physician diagnosed asthma was more specific but less sensitive[18]. In epidemiologic studies that estimate prevalence, a highly sensitive test is preferable; whereas, studies that estimate risk require more specific tests [19].

Excluding children with asthma-related symptoms from the sample increased the sensitivity (from 59% to 84%) and overall agreement (kappa increased from 0.60 to 0.75) between questionnaire and health claims diagnoses. Children with asthma-related symptoms may represent a substantial proportion of under-diagnosed asthma. This may have implications for cohort studies producing risk estimates[20] for putative risk factors for asthma incidence. Our findings support the practice of excluding children with asthma-related symptoms from the control group in epidemiological studies in order to decrease misclassification bias[20]. A limitation to this approach is the inflation of the odds ratio that occurs and the divergence of the odds ratio from the relative risk, making it impossible to calculate population-attributable risk from an exposure.

The diagnosis of asthma is problematic as subjects are often asymptomatic with normal physical examinations and normal pulmonary function tests between exacerbations[21]. This problem is compounded in children as they are often unable to do pulmonary function testing would might help to clarify the diagnosis. As such, the diagnosis often relies on symptom report which is subject to significant recall bias [22]. Given these limitations, health claims databases are a useful source of information as they capture data at the time of asthma exacerbation.

A larger issue in the study of asthma is that there is no accepted gold standard to confirm the diagnosis; therefore, studies evaluating diagnostic tests must use an imperfect reference standard. The accuracy of the test being evaluated is a measure of how closely it correlates with the reference standard. Given that the questionnaire and the health claims diagnosis measure different aspects of physician-diagnosed asthma, it is not surprising that the questionnaire has good validity against the health claims reference standard.

We have capitalized on the population-based data available through our universal health care system to validate the questionnaire asthma diagnosis in our T-CHEQ population. The results of this study may not be generalizable to a population that does not have equal access to health care.

This study is however the largest validation study reported to date and gives evidence that parental report on questionnaire is a highly specific method for identifying children with asthma in Canada.

Conclusions

Parental proxy report of asthma diagnosis by questionnaire has low sensitivity but high specificity as an asthma prevalence measure for epidemiological studies. Excluding children with asthma-related symptoms from non-asthma control groups will result in less misclassification bias.

Abbreviations

ISAAC:

International Study of Allergies and Asthma in Childhood

TCHEQ:

Toronto Child Health Evaluative Questionnaire

CIHI:

Canadian Institute for Health Information

OHIP:

Ontario Health Insurance Plan

ICD:

International Classification of Disease.

References

  1. Asher MI, Keil U, Anderson HR, Beasley R, Crane J, Martinez F, Mitchell EA, Pearce N, Sibbald B, Stewart AW, et al: International Study of Asthma and Allergies in Childhood (ISAAC): rationale and methods. Eur Respir J. 1995, 8 (3): 483-491. 10.1183/09031936.95.08030483.

    Article  CAS  PubMed  Google Scholar 

  2. Worldwide variations in the prevalence of asthma symptoms: the International Study of Asthma and Allergies in Childhood (ISAAC). Eur Respir J. 1998, 12 (2): 315-335.

  3. Celedon JC, Soto-Quiros ME, Silverman EK, Hanson L, Weiss ST: Risk factors for childhood asthma in Costa Rica. Chest. 2001, 120 (3): 785-790. 10.1378/chest.120.3.785.

    Article  CAS  PubMed  Google Scholar 

  4. Garcia-Marcos L, Garcia-Hernandez G, Morales Suarez-Varela M, Batlles Garrido J, Castro-Rodriguez JA: Asthma attributable to atopy: does it depend on the allergen supply?. Pediatr Allergy Immunol. 2007, 18 (3): 181-187. 10.1111/j.1399-3038.2006.00507.x.

    Article  PubMed  Google Scholar 

  5. Gehring U, Strikwold M, Schram-Bijkerk D, Weinmayr G, Genuneit J, Nagel G, Wickens K, Siebers R, Crane J, Doekes G, et al: Asthma and allergic symptoms in relation to house dust endotoxin: Phase Two of the International Study on Asthma and Allergies in Childhood (ISAAC II). Clin Exp Allergy. 2008, 38 (12): 1911-1920. 10.1111/j.1365-2222.2008.03087.x.

    Article  CAS  PubMed  Google Scholar 

  6. Beasley RW, Clayton TO, Crane J, Lai CK, Montefort SR, von Mutius E, Stewart AW: Acetaminophen Use and Risk of Asthma, Rhinoconjunctivitis and Eczema in Adolescents: ISAAC Phase Three. Am J Respir Crit Care Med. 2010

    Google Scholar 

  7. Prasad A, Langford B, Stradling JR, Ho LP: Exhaled nitric oxide as a screening tool for asthma in school children. Respiratory medicine. 2006, 100 (1): 167-173. 10.1016/j.rmed.2005.03.039.

    Article  PubMed  Google Scholar 

  8. Tu K, Mitiku T, Lee DS, Guo H, Tu JV: Validation of physician billing and hospitalization data to identify patients with ischemic heart disease using data from the Electronic Medical Record Administrative data Linked Database (EMRALD). Can J Cardiol. 2010, 26 (7): e225-228. 10.1016/S0828-282X(10)70412-8.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Lopushinsky SR, Covarrubia KA, Rabeneck L, Austin PC, Urbach DR: Accuracy of administrative health data for the diagnosis of upper gastrointestinal diseases. Surg Endosc. 2007, 21 (10): 1733-1737. 10.1007/s00464-006-9136-1.

    Article  CAS  PubMed  Google Scholar 

  10. To T, Dell S, Dick PT, Cicutto L, Harris JK, MacLusky IB, Tassoudji M: Case verification of children with asthma in Ontario. Pediatr Allergy Immunol. 2006, 17 (1): 69-76. 10.1111/j.1399-3038.2005.00346.x.

    Article  PubMed  Google Scholar 

  11. To T: Defining asthma in children for surveillance. American Journal of Respiratory & Critical Care Medicine. 2004, 169 (7): A383-

    Google Scholar 

  12. Dell SD, Foty RG, Gilbert NL, Jerret M, To T, Walter SD, Stieb DM: Asthma and allergic disease prevalence in a diverse sample of Toronto school children: Results from the Toronto Child Health Evaluation Questionnaire (T-CHEQ) Study. Can Respir J. 2010, 17 (1): e1-6.

    Article  PubMed  PubMed Central  Google Scholar 

  13. McGinn T, Wyer PC, Newman TB, Keitz S, Leipzig R, For GG: Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic). CMAJ. 2004, 171 (11): 1369-1373. 10.1503/cmaj.1031981.

    Article  PubMed  PubMed Central  Google Scholar 

  14. de Marco R, Cerveri I, Bugiani M, Ferrari M, Verlato G: An undetected burden of asthma in Italy: the relationship between clinical and epidemiological diagnosis of asthma. Eur Respir J. 1998, 11 (3): 599-605.

    CAS  PubMed  Google Scholar 

  15. Remes ST, Korppi M, Remes K, Pekkanen J: Prevalence of asthma at school age: a clinical population-based study in eastern Finland. Acta Paediatrica. 1996, 85 (1): 59-63. 10.1111/j.1651-2227.1996.tb13891.x.

    Article  CAS  PubMed  Google Scholar 

  16. Hederos CA, Hasselgren M, Hedlin G, Bornehag CG: Comparison of clinically diagnosed asthma with parental assessment of children's asthma in a questionnaire. Pediatric Allergy & Immunology. 2007, 18 (2): 135-141. 10.1111/j.1399-3038.2006.00474.x.

    Article  Google Scholar 

  17. Cerveri I, Bruschi C, Ricciardi M, Zocchi L, Zoia MC, Rampulla C: Epidemiological diagnosis of asthma: methodological considerations of prevalence evaluation. Eur J Epidemiol. 1987, 3 (2): 202-205.

    Article  CAS  PubMed  Google Scholar 

  18. Jenkins MA, Clarke JR, Carlin JB, Robertson CF, Hopper JL, Dalton MF, Holst DP, Choi K, Giles GG: Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma. International journal of epidemiology. 1996, 25 (3): 609-616. 10.1093/ije/25.3.609.

    Article  CAS  PubMed  Google Scholar 

  19. Pekkanen J, Pearce N: Defining asthma in epidemiological studies. Eur Respir J. 1999, 14 (4): 951-957. 10.1034/j.1399-3003.1999.14d37.x.

    Article  CAS  PubMed  Google Scholar 

  20. Pekkanen J, Sunyer J, Chinn S: Nondifferential disease misclassification may bias incidence risk ratios away from the null. Journal of Clinical Epidemiology. 2006, 59 (3): 281-289. 10.1016/j.jclinepi.2005.07.013.

    Article  PubMed  Google Scholar 

  21. Bacharier LB, Strunk RC, Mauger D, White D, Lemanske RF, Sorkness CA: Classifying asthma severity in children: mismatch between symptoms, medication use, and lung function. Am J Respir Crit Care Med. 2004, 170 (4): 426-432. 10.1164/rccm.200308-1178OC.

    Article  PubMed  Google Scholar 

  22. Brogger J, Eagan T, Eide GE, Bakke P, Gulsvik A: Bias in retrospective studies of trends in asthma incidence. Eur Respir J. 2004, 23 (2): 281-286. 10.1183/09031936.03.00041103.

    Article  CAS  PubMed  Google Scholar 

Pre-publication history

Download references

Acknowledgements

This work was supported by Health Canada (reference #4500171915).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sharon D Dell.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

SDD, TT, DS conceived of the study and participated in its design and coordination. RF cleaned and prepared data for linkage procedures. CLY performed the literature review and prepared the first draft of the manuscript. All authors read and approved of the manuscript.

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Yang, C.L., To, T., Foty, R.G. et al. Verifying a questionnaire diagnosis of asthma in children using health claims data. BMC Pulm Med 11, 52 (2011). https://doi.org/10.1186/1471-2466-11-52

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1471-2466-11-52

Keywords