- Research article
- Open Access
Verifying a questionnaire diagnosis of asthma in children using health claims data
BMC Pulmonary Medicine volume 11, Article number: 52 (2011)
Childhood asthma prevalence is widely measured by parental proxy report of physician-diagnosed asthma in questionnaires. Our objective was to validate this measure in a North American population.
The 2884 study participants were a subsample of 5619 school children aged 5 to 9 years from 231 schools participating in the Toronto Child Health Evaluation Questionnaire study in 2006. We compared agreement between "questionnaire diagnosis" and a previously validated "health claims data diagnosis". Sensitivity, specificity and kappa were calculated for the questionnaire diagnosis using the health claims diagnosis as the reference standard.
Prevalence of asthma was 15.7% by questionnaire and 21.4% by health claims data. Questionnaire diagnosis was insensitive (59.0%) but specific (95.9%) for asthma. When children with asthma-related symptoms were excluded, the sensitivity increased (83.6%), and specificity remained high (93.6%).
Our results show that parental report of asthma by questionnaire has low sensitivity but high specificity as an asthma prevalence measure. In addition, children with "asthma-related symptoms" may represent a large fraction of under-diagnosed asthma and they should be excluded from the inception cohort for risk factor studies.
Parental proxy report of physician-diagnosed asthma (questionnaire diagnosis) is the standard measure of childhood asthma prevalence used in national health surveys and epidemiological studies. The International Study of Allergies and Asthma in Childhood (ISAAC) questionnaire is the current gold standard for ascertaining asthma outcomes in epidemiologic studies . The "ever wheeze" question is often used to compare asthma prevalence between countries. For international studies, a symptom-based definition is less subject to bias than a diagnosis-based definition. However, for national studies that evaluate risk factors, a more specific definition is often desired [3–5]. In these studies, lifetime asthma is measured by affirmative response to the question: "Has your child ever had asthma?" and is further defined to be "Doctor diagnosed asthma" if there is an affirmative response to the question "Has this been diagnosed by a doctor?" Despite its widespread use in studies assessing asthma prevalence, risk factors ,and diagnostic tests  this questionnaire based asthma diagnosis has not been validated in a North American population.
Canada has a universal health care system in which all Canadians have equal access to physician and hospital services. Health claims data on all patient encounters is collected for administrative purposes although in Ontario there is currently no centralized database on prescription medication. These databases have been found to be accurate compared to chart abstraction for diseases such as ischemic heart disease, esophagitis as well as asthma. Recently, an algorithm has been developed to identify children with asthma from health claims data in the province of Ontario. This algorithm, has been shown to have 91.4% sensitivity and 82.9% specificity for correctly identifying asthma when compared to expert consensus diagnosis of asthma[10, 11].
The objective of this research was to assess agreement between questionnaire based parental proxy report of physician diagnosed asthma in children (hereafter referred to as "questionnaire diagnosis") and asthma diagnosed by analyzing health claims data (hereafter referred to as "health claims diagnosis") in a population-based sample of urban Canadian school children.
Participants were recruited from The Toronto Child Health Evaluation Questionnaire (T-CHEQ) study. This study used ISAAC methodology to recruit a population based sample of 5619 children (aged 5 to 9 years) from 231 Toronto public schools between January and May of 2006. The demographic characteristics of these children closely resembled census data and the prevalence of asthma outcomes closely resembled national health survey data. Detailed methods for sampling and recruitment are published elsewhere. All subjects that participated in the T-CHEQ study were asked at the time of the initial study if they would consent to have their child's questionnaire linked to health claims data for research purposes and those that agreed were included in this study.
This is a diagnostic validation study, comparing agreement between questionnaire asthma diagnosis (using data from the cross-sectional T-CHEQ study) and health claims diagnosis (using cohort data from lifetime health claims administrative databases).
Questionnaire based Asthma diagnosis
Asthma diagnosis was identified from the T-CHEQ study sample by affirmative responses to the questions "Has your child ever had asthma?" and "Was this diagnosed by a doctor?" Those that reported non-physician diagnosed asthma were excluded from the analysis. Non-asthma controls did not report doctor diagnosed asthma. These controls were further categorized into those with "Asthma-related symptoms" if they reported a yes response to either "Has your child ever had wheezing" or "In the past 12 months has your child had a dry cough at night, apart from a cough associated with a cold or chest infection?".
Health Claims Asthma Diagnosis (Reference Standard)
Health claims data between March 31, 1997 and March 31, 2006 (i.e. the child's lifetime) from two Ontario health care administrative databases were used: (1) the Canadian Institute for Health Information (CIHI) discharge abstract database for inpatient services and (2) The Ontario Health Insurance Plan (OHIP) for ambulatory and emergency services. Both of these databases contain diagnostic codes based on the International Classification of Disease (ICD)-9 or 10. A claim for asthma was identified by the ICD- 9 code 493 for claims up to March 31, 2002 and ICD-10 codes J45 and J46 identified asthma in subsequent years. The CIHI database currently allows up to 25 diagnostic codes (prior to 2002, 16 diagnostic codes were allowed) and if any of these was for asthma, the hospitalization was included. The OHIP database allows one diagnostic code per visit. Only one claim per physician per day per patient was allowed. The health claims databases were also linked to the Registered Persons Database which contains mortality and demographic data to ensure that the subjects had lived within the province since birth. A unique personal identifier (the scrambled Health Card Number) included in each database permits the linkage of a child's records across all databases and time while preserving patient confidentiality.
Prevalent asthma cases were defined by a previously validated algorithm as follows: at least one hospitalization for asthma at any time during the child's life or two separate ambulatory or emergency room visits for asthma within a two year time frame. This algorithm was previously found to have optimal diagnostic parameters using an expert consensus diagnosis from chart abstraction[10, 11].
The T-CHEQ participants were anonymously linked to the health claims databases through their reported Health Insurance Number. A matching date of birth in the two databases was also needed for the link to be considered valid. All data linkage and analysis related to this study was completed within the secure confines of the Institute of Clinical and Evaluative Sciences in Toronto, Ontario.
"Questionnaire diagnosis" and "health claims diagnosis" were compared in two by two tables. Sensitivity and specificity of the questionnaire diagnosis were calculated using health claims diagnosis as the gold standard. In order to test the potential misclassification bias that could occur by including children with asthma-related symptoms in the non-asthma control group, the children with asthma-related symptoms were removed from the sample and sensitivities and specificities were recalculated. Additional sensitivity analyses were conducted by modifying the health claims data algorithm (increasing the time frame for incident asthma ambulatory claims from 2 to 3 years or including emergency visit data from an additional database). We also calculated agreement (kappa) between questionnaire and health claims diagnosis.
Parents of children in this study gave informed consent for participation in this research by filling in the voluntary T-CHEQ questionnaire and additionally agreeing to participate in the data linkage. This study was approved by the Research Ethics Board of the Hospital for Sick Children in Toronto.
From the original TCHEQ cohort, 2884 (51.32%) gave permission to link to health claims data. Respondents did not differ from non-respondents in terms of asthma prevalence or gender, however they were more likely to be in a higher income group (42.13% versus 31.40%) and have post-graduate education (51.96% versus 40.09%), and were less likely to report no physician visits in the past year (24.85% versus 29.57%) (Table 1). Data linkage was successfully achieved in 2782 of the respondents (96.46%).
In our study sample, 437 children were identified with questionnaire diagnosis of asthma (prevalence of 15.71%) while 586 children were identified with health claims diagnosis of asthma (prevalence of 21.06%) (Table 2). The vast majority of subjects with ever asthma (defined by the question "Has your child ever had asthma?") had physician diagnosed asthma (also responded affirmatively to the question "Was this diagnosed by a doctor?") (prevalence of asthma 16.41% and 15.71% respectively).
The questionnaire diagnosis had a sensitivity of 59.04% and a specificity of 95.86% for detecting asthma using the health claims diagnosis as the reference standard.
Of the 2435 non-asthma controls, 854 (35%) had asthma-related symptoms. When these children were removed from the sample (table 3), the sensitivity and specificity of the questionnaire diagnosis were 83.57% and 93.56% respectively.
The sensitivity analysis performed using modified algorithms for definitions of health claims diagnosis did not produce any significantly different findings (data not shown).
We observed moderate agreement (kappa = 0.60) between questionnaire asthma definitions and health claims asthma definitions. Good agreement (kappa = 0.75) was observed when those with asthma-related symptoms were excluded.
Our findings concur with other literature that suggests that questionnaire asthma diagnosis is specific but not sensitive for asthma[14–17]. As expected, compared with questionnaires that use a definition of "wheezing in the last twelve months" to define the population with asthma, our definition of physician diagnosed asthma was more specific but less sensitive. In epidemiologic studies that estimate prevalence, a highly sensitive test is preferable; whereas, studies that estimate risk require more specific tests .
Excluding children with asthma-related symptoms from the sample increased the sensitivity (from 59% to 84%) and overall agreement (kappa increased from 0.60 to 0.75) between questionnaire and health claims diagnoses. Children with asthma-related symptoms may represent a substantial proportion of under-diagnosed asthma. This may have implications for cohort studies producing risk estimates for putative risk factors for asthma incidence. Our findings support the practice of excluding children with asthma-related symptoms from the control group in epidemiological studies in order to decrease misclassification bias. A limitation to this approach is the inflation of the odds ratio that occurs and the divergence of the odds ratio from the relative risk, making it impossible to calculate population-attributable risk from an exposure.
The diagnosis of asthma is problematic as subjects are often asymptomatic with normal physical examinations and normal pulmonary function tests between exacerbations. This problem is compounded in children as they are often unable to do pulmonary function testing would might help to clarify the diagnosis. As such, the diagnosis often relies on symptom report which is subject to significant recall bias . Given these limitations, health claims databases are a useful source of information as they capture data at the time of asthma exacerbation.
A larger issue in the study of asthma is that there is no accepted gold standard to confirm the diagnosis; therefore, studies evaluating diagnostic tests must use an imperfect reference standard. The accuracy of the test being evaluated is a measure of how closely it correlates with the reference standard. Given that the questionnaire and the health claims diagnosis measure different aspects of physician-diagnosed asthma, it is not surprising that the questionnaire has good validity against the health claims reference standard.
We have capitalized on the population-based data available through our universal health care system to validate the questionnaire asthma diagnosis in our T-CHEQ population. The results of this study may not be generalizable to a population that does not have equal access to health care.
This study is however the largest validation study reported to date and gives evidence that parental report on questionnaire is a highly specific method for identifying children with asthma in Canada.
Parental proxy report of asthma diagnosis by questionnaire has low sensitivity but high specificity as an asthma prevalence measure for epidemiological studies. Excluding children with asthma-related symptoms from non-asthma control groups will result in less misclassification bias.
International Study of Allergies and Asthma in Childhood
Toronto Child Health Evaluative Questionnaire
Canadian Institute for Health Information
Ontario Health Insurance Plan
International Classification of Disease.
Asher MI, Keil U, Anderson HR, Beasley R, Crane J, Martinez F, Mitchell EA, Pearce N, Sibbald B, Stewart AW, et al: International Study of Asthma and Allergies in Childhood (ISAAC): rationale and methods. Eur Respir J. 1995, 8 (3): 483-491. 10.1183/09031936.95.08030483.
Worldwide variations in the prevalence of asthma symptoms: the International Study of Asthma and Allergies in Childhood (ISAAC). Eur Respir J. 1998, 12 (2): 315-335.
Celedon JC, Soto-Quiros ME, Silverman EK, Hanson L, Weiss ST: Risk factors for childhood asthma in Costa Rica. Chest. 2001, 120 (3): 785-790. 10.1378/chest.120.3.785.
Garcia-Marcos L, Garcia-Hernandez G, Morales Suarez-Varela M, Batlles Garrido J, Castro-Rodriguez JA: Asthma attributable to atopy: does it depend on the allergen supply?. Pediatr Allergy Immunol. 2007, 18 (3): 181-187. 10.1111/j.1399-3038.2006.00507.x.
Gehring U, Strikwold M, Schram-Bijkerk D, Weinmayr G, Genuneit J, Nagel G, Wickens K, Siebers R, Crane J, Doekes G, et al: Asthma and allergic symptoms in relation to house dust endotoxin: Phase Two of the International Study on Asthma and Allergies in Childhood (ISAAC II). Clin Exp Allergy. 2008, 38 (12): 1911-1920. 10.1111/j.1365-2222.2008.03087.x.
Beasley RW, Clayton TO, Crane J, Lai CK, Montefort SR, von Mutius E, Stewart AW: Acetaminophen Use and Risk of Asthma, Rhinoconjunctivitis and Eczema in Adolescents: ISAAC Phase Three. Am J Respir Crit Care Med. 2010
Prasad A, Langford B, Stradling JR, Ho LP: Exhaled nitric oxide as a screening tool for asthma in school children. Respiratory medicine. 2006, 100 (1): 167-173. 10.1016/j.rmed.2005.03.039.
Tu K, Mitiku T, Lee DS, Guo H, Tu JV: Validation of physician billing and hospitalization data to identify patients with ischemic heart disease using data from the Electronic Medical Record Administrative data Linked Database (EMRALD). Can J Cardiol. 2010, 26 (7): e225-228. 10.1016/S0828-282X(10)70412-8.
Lopushinsky SR, Covarrubia KA, Rabeneck L, Austin PC, Urbach DR: Accuracy of administrative health data for the diagnosis of upper gastrointestinal diseases. Surg Endosc. 2007, 21 (10): 1733-1737. 10.1007/s00464-006-9136-1.
To T, Dell S, Dick PT, Cicutto L, Harris JK, MacLusky IB, Tassoudji M: Case verification of children with asthma in Ontario. Pediatr Allergy Immunol. 2006, 17 (1): 69-76. 10.1111/j.1399-3038.2005.00346.x.
To T: Defining asthma in children for surveillance. American Journal of Respiratory & Critical Care Medicine. 2004, 169 (7): A383-
Dell SD, Foty RG, Gilbert NL, Jerret M, To T, Walter SD, Stieb DM: Asthma and allergic disease prevalence in a diverse sample of Toronto school children: Results from the Toronto Child Health Evaluation Questionnaire (T-CHEQ) Study. Can Respir J. 2010, 17 (1): e1-6.
McGinn T, Wyer PC, Newman TB, Keitz S, Leipzig R, For GG: Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic). CMAJ. 2004, 171 (11): 1369-1373. 10.1503/cmaj.1031981.
de Marco R, Cerveri I, Bugiani M, Ferrari M, Verlato G: An undetected burden of asthma in Italy: the relationship between clinical and epidemiological diagnosis of asthma. Eur Respir J. 1998, 11 (3): 599-605.
Remes ST, Korppi M, Remes K, Pekkanen J: Prevalence of asthma at school age: a clinical population-based study in eastern Finland. Acta Paediatrica. 1996, 85 (1): 59-63. 10.1111/j.1651-2227.1996.tb13891.x.
Hederos CA, Hasselgren M, Hedlin G, Bornehag CG: Comparison of clinically diagnosed asthma with parental assessment of children's asthma in a questionnaire. Pediatric Allergy & Immunology. 2007, 18 (2): 135-141. 10.1111/j.1399-3038.2006.00474.x.
Cerveri I, Bruschi C, Ricciardi M, Zocchi L, Zoia MC, Rampulla C: Epidemiological diagnosis of asthma: methodological considerations of prevalence evaluation. Eur J Epidemiol. 1987, 3 (2): 202-205.
Jenkins MA, Clarke JR, Carlin JB, Robertson CF, Hopper JL, Dalton MF, Holst DP, Choi K, Giles GG: Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma. International journal of epidemiology. 1996, 25 (3): 609-616. 10.1093/ije/25.3.609.
Pekkanen J, Pearce N: Defining asthma in epidemiological studies. Eur Respir J. 1999, 14 (4): 951-957. 10.1034/j.1399-3003.1999.14d37.x.
Pekkanen J, Sunyer J, Chinn S: Nondifferential disease misclassification may bias incidence risk ratios away from the null. Journal of Clinical Epidemiology. 2006, 59 (3): 281-289. 10.1016/j.jclinepi.2005.07.013.
Bacharier LB, Strunk RC, Mauger D, White D, Lemanske RF, Sorkness CA: Classifying asthma severity in children: mismatch between symptoms, medication use, and lung function. Am J Respir Crit Care Med. 2004, 170 (4): 426-432. 10.1164/rccm.200308-1178OC.
Brogger J, Eagan T, Eide GE, Bakke P, Gulsvik A: Bias in retrospective studies of trends in asthma incidence. Eur Respir J. 2004, 23 (2): 281-286. 10.1183/09031936.03.00041103.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2466/11/52/prepub
This work was supported by Health Canada (reference #4500171915).
The authors declare that they have no competing interests.
SDD, TT, DS conceived of the study and participated in its design and coordination. RF cleaned and prepared data for linkage procedures. CLY performed the literature review and prepared the first draft of the manuscript. All authors read and approved of the manuscript.
About this article
Cite this article
Yang, C.L., To, T., Foty, R.G. et al. Verifying a questionnaire diagnosis of asthma in children using health claims data. BMC Pulm Med 11, 52 (2011). https://doi.org/10.1186/1471-2466-11-52
- Health Claim
- Asthma Prevalence
- Asthma Diagnosis
- Parental Proxy Report