The registry serves 23 health care sites that provide the great majority of the pediatric primary care for Boston, an urban setting with a 2000 population of about 116,000 children < 18 years old . Only one practice site is private; all others are either in community health centers or in academic medical centers. This study consisted of two separate evaluations of the ability to match patients' medical records to registry records and the accuracy of immunization records in the registry. The first evaluation focused on pediatric primary care sites that were active registry participants, appraising the registry's current performance. This evaluation thus reflects the cumulative impact of all methods used for immunization data transfer to the registry over the history of the registry, through the time of the evaluation. The second evaluation focused on sites that had implemented a comprehensive EMR within 2–3 years before the evaluation. This second evaluation thus appraises the expected registry performance with direct transfer of immunization data from the EMR to the registry. The study was approved by institutional review boards serving all participating primary care sites, the Boston Public Health Commission, and the Boston University Schools of Public Health and Medicine. Only Boston Public Health Commission registry managers had access to individual registry records; registry data provided to Boston University investigators were either aggregated or de-identified.
Collection of data from primary care providers
Pediatric primary care sites were identified by registry managers as 1) current registry users with good overall data integrity and completeness or 2) future direct-EMR-transfer users – former registry users that had temporarily stopped transferring data to the registry to allow implementation of the EMR with direct data transfer. All sites were invited to participate in the data quality review. Data were collected from August 2004 through February 2005 from sites that enrolled. For each primary care site, a random sample of pediatric records from current patients aged 0–10 and 11–17 years was chosen for review from the site's list of patients. For sites that reported serving < 5000 pediatric patients, 25 charts were chosen within each age group; for larger sites, 50 records were chosen within each age group. All components of each subject's medical record, including all volumes of paper records and all electronic records, were reviewed. For each subject, information on all immunizations containing diphtheria and tetanus toxoids was recorded; both those with and without pertussis antigens were included. We refer to these immunizations as "pertussis-related" hereafter. Data included the immunization date, formulation (DTwP, DTaP, DT, or Td; Tdap had not been introduced at the time of the study), manufacturer, and lot number. Additional data for each subject included 1) identifying information that might assist in matching records to the registry, including name, sex, date of birth, address, and mother's name; 2) dates of the patient's first visit to and first immunization at the primary care site; 3) for 11–17-year-olds, dates of MMR, hepatitis B, and varicella immunizations. When immunization data were recorded more than once in a subject's medical record, the data entered closest to the immunization date were considered to be the most accurate.
Matching subjects to registry records
For primary care sites that were current registry users, subjects were matched to registry records by registry managers. A match was defined as occurring when a subject's provider record matched exactly one registry record with definite correspondence of at least name, sex, and date of birth. The spelling of names was not required to be identical. When more than one registry record existed for a given name, sex, and date of birth, other identifying data including immunization dates could be used to determine whether there was a match. For primary care sites that were future direct-EMR-transfer users, identifying data in the EMR would be expected to be identical to directly transferred registry data. Therefore, failure to match would be expected to occur only if a patient's paper medical record had not been updated into an EMR during the implementation of the EMR system. Therefore, for subjects from these sites, failure to match was considered to occur only when a subject did not have an EMR.
Evaluation of immunization discrepancies between provider and registry records
For subjects from sites that were current registry users and who matched a registry record, registry managers evaluated whether discrepancies in immunization data existed between the provider record and the registry record. For each pertussis-related immunization, date discrepancies were categorized as 1–29 days difference, 30–364 days difference, or ≥ 1 year difference; formulation discrepancies were noted; and, for provider records with manufacturer and lot number information, any discrepancies in these fields were noted. Also, registry managers noted whether any discrepancies would have affected overall agreement between the provider record and the registry record as to whether the subject was currently up-to-date for pertussis-related immunization. For 11–17-year-old subjects, provider records of MMR, hepatitis B, and varicella immunization dates were compared to registry records using the same methods. To evaluate the sensitivity of the registry managers' evaluation of immunization discrepancies, artificial discrepancies were created in 10% of provider records and the detection of these discrepancies was tracked.
For subjects from primary care sites that were future direct-EMR-transfer users, immunization data in the EMR was considered to represent the data that would exist in the registry; the current EMR served as a proxy for the registry. Therefore, an immunization discrepancy was considered to have occurred when data in the EMR differed from data in the entire medical record, including all original volumes of the original paper record.
We calculated frequencies of record matching and, among matched records, frequencies of discrepancies in immunization information for current user sites as a group, for future direct-EMR-transfer sites as a group, and for each site individually. We categorized both current age and age at first visit to the provider site into two categories – 0–10-years-old and 11–17-years-old. We then used the chi-square statistic to test differences in matching frequency. We used logistic regression to examine the independent contributions of current age, age at first visit to the provider site, and sex to 1) ability to match a registry record and 2) among matched records, discrepancy in whether the subject was up-to-date for pertussis-related immunization, constructing separate models for subjects from current user sites and from future direct-EMR-transfer sites. We analyzed the data with Excel 2003 and SAS version 9.1 (SAS Institute, Inc., Cary, NC).