- Research article
- Open Access
- Open Peer Review
Properties of a short questionnaire for assessing Primary Care experiences for children in a population survey
BMC Public Health volume 11, Article number: 285 (2011)
The Primary Care Assessment Tool (PCAT) is an interesting set of tools for primary care research. A very short version could inform policy makers about consumer experiences with primary care (PC) through health surveys. This work aimed to investigate the validity and reliability of a selection of items from the child short edition (CS) of the PCAT.
A 24 item questionnaire permitted the identification of a regular source of care and the assessment of the key attributes of first contact, ongoing care over time, coordination, services available and services received (comprehensiveness), and cultural competence. Structural validity, reliability, and construct validity were assessed using responses from 2,200 parents of a representative sample of the population aged 0 to 14 years in Catalonia (Spain) who participated in the 2006 Health Survey. Structural validity was analyzed using exploratory and confirmatory factor analyses and reliability was assessed using Cronbach's alpha. Construct validity was assessed using linear regression analysis between PC experience scores and a measure of overall user satisfaction with healthcare services.
A total of 2,095 (95.2%) parents provided useable responses on PC. After Confirmatory Factor Analysis (CFA), the best fitting model was a 5-factor model in which the original dimensions of first contact and ongoing care were collapsed into one. The CFA also showed a second order factor onto which all domains except services available loaded (root mean square error of approximation = 0.000; comparative fit index = 1.00). Cronbach's alpha values for one of the original scales (first-contact) was poor (alpha < 0.50), but improved using the modified factor structure (alpha > 0.70). Scores on the scales were correlated with satisfaction with healthcare services (p < 0.01), thereby providing some preliminary evidence of construct validity.
This very short questionnaire obtained from the PCAT-CE yields information about five attributes of PC and a summary score. It has shown evidence of validity and reliability for judgments about experiences with primary care overall. If space on surveys is at a premium, the instrument could be useful as a measure of PC experiences.
The importance of primary care (PC) in making health care delivery more efficient and in tackling health inequalities has been discussed at a political level worldwide for at least the last three decades . Several countries have reformed their health system with a particular emphasis on strengthening primary care. Public health researchers have made considerable efforts to assess the quality of primary care and the extent of improvements. Interest in a broader theoretical model of PC has also grown, and it has become clear that there is a need for research tools to measure important attributes of primary care from the user's point of view [2, 3].
Theoretical models of primary care have stressed its structural and organizational aspects [4, 5] and other attributes essential to primary contact for users and their families, focusing on the user perspective [6, 7]. One of the most widely acknowledged definitions of the quality of PC services is "the provision of accessible and integrated services addressing a large majority of health care needs, developing a sustained relationship with patients, and practicing in the context of family and community" . The Primary Care Assessment Tools (PCAT), developed in the United States (US), addresses these issues .
The PCAT is a set of questionnaires consisting of consumer-client surveys, facility surveys, and provider surveys. Their development provided a set of tools for PC evaluation that have become increasingly widely used in several countries [10–13]. The different versions include items designed to collect information on four core and three ancillary domains of PC. The core attributes are : first contact, i.e. assessment of PC's role as the entry point to the health care system except in emergency situations; ongoing care, i.e. maintenance of person-focused care over time; comprehensiveness, which deals with PC's ability to make available and provide a wide range of services, including but not limited to preventive care, in response to prevalent health needs; and coordination, i.e. to support and integrate the care of problems addressed elsewhere, either by practitioners or medical records. PCAT also addresses the extent to which PC focuses on the health of individuals within the context of family (family orientation), is able to tackle community health problems (community orientation), and can deal with the need to establish relationships with people from different social groups (cultural competence).
Spain initiated substantial health care reforms in 1986, by legislating for universal tax-financed services, increasing the proportion of total health expenditure which is publicly financed,  and making the strengthening of primary care teams a central part of its strategy in primary care. The state-run health service provides free access to services, but approximately 10% of adult people and one third of the infant population pay for additional private healthcare coverage. Autonomous Communities have full autonomy in the health-care sector in Spain. In Catalonia, in the north-east of Spain, health interview surveys have been conducted periodically since 1983 (Barcelona city) and 1994 (Catalonia) to evaluate the population's health status and its determinants, and to assess the performance of healthcare services. In the 2006 edition of the Catalan Health Interview Survey, the intention to assess Primary Health Care led to interest in implementing the PCAT. Since the expanded version of the PCAT for children (PCAT-CE) or the short version (PCAT-CS) could not be included due to limitations in the length of the survey, several items from the PCAT-CS were selected to provide information on primary care. Given that use of this subset of items represented a substantial modification to the original instrument, it was considered necessary to re-assess its psychometric characteristics. The research question addressed in this paper is whether the selected items provide a valid and reliable measure of parent experiences with PC.
Sample and setting
The 2006 Catalan Health Interview Survey was a cross-sectional study carried out in a representative sample of non-institutionalized residents of Catalonia. Sample selection used a multi-stage design. First, municipalities were selected according to number of inhabitants and, second, individuals were selected based on age and sex distribution from the population registry of the Statistical Institute of Catalonia (IDESCAT). The sample size for the population under 15 years was established at 2,200 infants and children. Main cases and five replacements were initially selected. The sample size was reached with 65% of main cases and 22% of first replacement cases; the remainders were mostly second replacement cases. Reasons for replacement were because parents refused the interview or were not found at the available address. The survey questionnaire was administered during home-based interviews with a proxy respondent, preferably the child's usual caregiver.
The instrument to assess PC in Spain
The original child edition of the short, consumer version of the PCAT contains 44 items (PCAT-CS) distributed over 7 domains . Five initial questions are used to identify the source of Primary Care. First contact accessibility and utilization (first contact domain), longitudinal interpersonal relationships (ongoing care domain), coordination of services (coordination domain), comprehensiveness of services available and received (comprehensiveness domain) are assessed through questions on how confidant the respondent is about their provision by the primary care source. Three ancillary domains (family centeredness, community orientation, and cultural competence) are also assessed.
To include the PCAT in the 2006 Catalan Health Interview Survey questionnaire for 0 to 14 year olds, it was necessary to select a subset of items due to limitations on space. Based on the PCAT-CS, the local working group together with one of the authors of the original version agreed on a very short version that was considered to best represent the thrust of the original questionnaire. This compromise led to the exclusion of the Community and Family Orientation domains, and the selection of a minimum of two items from the remaining domains based on their interest in covering a broader range of aspects of primary care performance. The final version included twenty-four items adapted to the Spanish context, since the Health System is homogeneous in all the Autonomous Communities in Spain. Two initial questions ask if children have a regular source of primary care or, if not, about the last general practitioner or pediatrician that they visited. Three questions characterize the regular source of care. The remaining 19 items cover five of the seven domains in the original questionnaire, including first contact (4 items, mostly addressing accessibility), ongoing care (3 items), coordination of services (a filter plus 2 items), comprehensiveness (4 items in the services available subdomain, and (3 items of the services received subdomain, one of which is only for adolescents), and cultural competence (2 items). Items and other text were translated into Catalan and Spanish, the main official languages spoken in Catalonia, following an internationally recommended process of cross-cultural adaptation to achieve semantic equivalence with the original version . Further details of the selection of items and the cross-cultural adaptation have been published elsewhere . The current analysis focuses only on the 17 items actually used to assess PC attributes across the full age-range for paediatric care. The 5 items used to identify and characterise the source of care were therefore excluded, together with the filter question before the coordination domain, and one item which only applies to adolescents.
All items covering the PC domains are answered on a 4-point Likert-type scale (1 = definitely not; 2 = probably not; 3 = probably yes; and 4 = definitely yes). Additional response options included "don't know" or "can't remember" . The score for each domain is computed as the mean value for all items in that domain, and can range between 1 and 4.
The Catalan Health Interview Survey is an observational health survey conducted by the Department of Health of Catalonia as part of routine governmental statistics gathering. The survey complies with all relevant national legislation on the protection and processing of personal data. Data are not openly available but can be obtained on request from the Catalan Department of Health, and the entire questionnaire can be accessed in its website .
Demographic and health characteristics were described for the sub-sample of the Catalan Health Interview Survey used in the present analysis. As those who did not identify a regular source of care were excluded from the remainder of the analysis, they were compared with those included using chi-square to test for differences on demographic and health status variables.
Several factor analyses were carried out to evaluate the structural validity of this modified version of the PCAT. Factor analysis is a multivariate technique that allows the implementation of a statistical model that reproduces the item variance-covariance matrix and provides statistical information on model quality. In the first instance, we used exploratory factor analysis (EFA) to extract an increasing number of first order factors and account for observed variability without further restrictions on the model. Factors were extracted using diagonally weighted least squares (DWLS) on the item polychoric correlation matrix. Oblimin rotation was applied to allow for correlation between factors. We hypothesized that items would load onto domains in the same way as in the original questionnaire, thereby indicating the appropriateness of the underlying conceptual model for the Catalan population. Preliminary results from this initial factor analysis were further tested using Confirmatory Factor Analysis (CFA). In this case, certain restrictions were imposed on factor structure, and the presence of second order factors and the structural validity of a global score encompassing all domains were also tested. The unweighted least squares estimation method on the item polychoric correlation matrix was used to obtain p-values and standard errors robust for non-normal distributions. This method respects the ordered categorical nature of the items while providing robust estimates of factor loading. The model's overall goodness of fit was assessed using the root mean square error of approximation (RMSEA), and two incremental fit indices, the comparative fit index (CFI), and the Tucker Lewis-Index (TLI). A RMSEA under 0.06 was considered to indicate an excellent fit between the specified model and the data, whereas values of CFI and TLI were required to be over 0.95. The analyses were repeated separately for infants (0 to 5 years), children (6 to 11 years) and adolescents (12 to 14 years) to take into account possible structural variance across age groups.
Models with the best results in the EFA were studied in more depth using confirmatory factor analysis. Pearson correlation coefficients were used to determine the degree of correlation between domains; some degree of correlation between factors was expected.
The descriptive and reliability analysis was conducted based on both the questionnaire structure as administered and the new structure stemming from factor analysis. Score distributions for the 17 items of the PCAT domains was carried out by calculating the mean and standard deviation, the proportion of missing responses, the observed score range, and floor and ceiling effects (i.e. the proportion of cases with the worst and best possible score, respectively). The percentage of items which correlated higher with their hypothesized domain than with the other domains obtained in the EFA (scaling success rates) was calculated. Homogeneity between items in each factor derived from the factor analysis was assessed using Cronbach's alpha.
General satisfaction with services was used to assess construct validity. Satisfaction was analyzed by an item in the main survey questionnaire with four-point Likert-type response scale where 1 = very satisfied, and 4 = very dissatisfied . Validity was assessed by using individual linear regression models with each factor of the PCAT to predict the dissatisfaction score. We expected to find a moderate association between user satisfaction scores and scores derived for each factor of the PC experience scales. Associations were expected to be negative because satisfaction is scored inversely (1 = satisfied) to the PC experience scales (1 = negative).
SPSS 15.0 was used for the descriptive analysis and Mplus version 5.2 was used for the factorial and regression analyses.
A total of 2.091 parents identified a usual source of care (92.5%), and 4 others (2.7%) had visited a health professional during the previous year. Thus 2.095 parents responded about their experiences with primary care for their children. Sample distribution by age and sex was equivalent to that of the Catalan general population . The proportion of parents of adolescents who were excluded from analysis because they did not identify a usual source of care was higher than in those included in the study, as was the proportion of parents who declared that their child had a chronic condition (table 1). Other differences between these two groups by sex, educational level or type of health care coverage were not statistically significant.
The exploratory factor analysis was conducted to investigate several possible solutions. The goodness of fit of the models was acceptable for all solutions between one and six factors, and the hypothesis of overall scale unidimensionality was rejected (Chi2 = 46.69, gl = 12, p < .001). The model of the six original scales (first contact, ongoing care, coordination, comprehensiveness of services available, comprehensiveness services received, and cultural competence) showed good results, but Heywood cases were observed, and some items loaded onto more than one factor. Heywood cases appear whenever factor loading estimates exceed 1, which implies that unique factors have negative error variances. These cases are likely to occur when there are too many common factors or when there are too many factors to provide stable estimates with the available data. The solution which best fitted the conceptual model was a five factor model with an RMSEA of 0.000, a CFI of 1.00, a TLI of 1.03, and a Chi2 = 3.70 (gl = 6, p = 0.72). In this model, most items loaded highest on the domains they were assigned to in the conceptual model. The exception was the first factor, in which the first contact and ongoing care domains merged (table 2). Another model with acceptable goodness of fit was the most parsimonious, with two factors (RMSEA = 0.023; CFI = 0.95; TLI = 0.92; Chi2 = 21.28, gl = 12, p = 0.019). In this case, the majority of items were grouped within the first factor, whilst a second factor consisted of items measuring services available (comprehensiveness).
Confirmatory factor analysis was conducted for the two models showing the best results in the previous exploratory phase. The five-factor model generated a second order factor which could be defined as "experiences with the content of PC" (F6, see Figure 1) and onto which F1 (first contact accessibility and ongoing care), F2 (coordination), F3 (services received), and F4 (cultural competence) all loaded. Only the services available domain did not load onto this 2nd order factor (F5). This "five plus one" model showed excellent goodness of fit (RMSEA = 0.01, CFI = 0.98; TLI = 0.97; Chi2 = 14.50, gl = 11, p = 0.20) and loadings were high for all six factors. The bifactorial model also presented good results in the CFA (RMSEA = 0.02; CFI = 0.94; TLI = 0.92; Chi2 = 22.86, gl = 11, p = 0.02) with items loading onto factors in a similar fashion to that observed in the EFA and the five plus one model. The multi-group analysis conducted to study the invariance of the structure between age groups confirmed an equivalent factorial structure within the three age groups tested, for the five plus one (Chi2 = 75.0, gl = 53, p = 0.03), and the bifactorial (Chi2 = 67.77, gl = 53, p = 0.06) solutions.
Score distributions by domain and the results of the reliability analysis are shown in table 3. These analyses were carried out for both the six single scales following the structure of the adminstered instrument, and the two new factors obtained from the factorial analysis: the merged first contact accessibility & continuity, and the second order factor ("experience with the content of PC"). Completion rates were high, with no missing responses on any items, except for the coordination domain, which only people who had visited a specialist were required to answer. Means and standard deviations showed generally positive experiences with primary care and distributions were non-normal, with a marked ceiling effect in Coordination and Cultural Competence. Internal consistency was poor (alpha < 0.50) for the First Contact-Accessibility domain and moderate (alpha < 0. 57-0.70) in three of the original six (ongoing care, services available, and services provided). The merged scales resulting from factor analysis showed a lower ceiling effect and achieved satisfactory homogeneity (alpha > 0.70). Scaling success was not achieved in the ongoing care domain, but was satisfactory in the remaining domains (Table 3).
The PCAT scales showed the expected pattern of association with the overall satisfaction with health-care item, i.e. parents reporting dissatisfaction with healthcare services scored low in most of the PC experiences domains (table 4).
This study analyzed the structure, construct validity and reliability of a very short version of the PCAT for parents of children aged 0 - 14 years. This instrument was derived from the original short version developed by researchers in the US (PCAT-CS), and its seventeen items measure several attributes of PC, based on the questionnaire's original conceptual model, with acceptable reliability and validity.
An important finding from the study included the fact that items about ongoing care correlated highly with those measuring first contact accessibility, to the extent that they all loaded onto a single factor. The original PCAT 'ongoing care' domain contained two types of items: those concerning identification with a particular provider and those concerning the nature of the relationship. In the very abridged version used in this study, 2 of the 3 items from the subdomain of interpersonal relationships were associated, making them similar in concept to the items on access in the first-contact subdomain. There is thus a difference with the original instrument, since the smaller number of items and the traits captured by them cannot collect equal information addressing the nature of the relationship, i.e., person focused care over time. Apart from this limitation, the analysis of structural validity indicated that the items selected adequately represent the original conceptual multidimensional model because data fitted well to the remaining 4 domains of coordination, services available, services provided, and cultural competence.
The bifactorial model was also confirmed as an alternative measure of two concepts, with good statistical fit. 'Services available' (a list of characteristics of the source of care that might be needed at some time) is different from services received and, hence, not strictly an 'experiences' measure. Moreover, the second order factor resulting from the confirmatory factor and its reliability analysis underlined the possibility of gathering the information on a unique scale. This option might be useful when a single global measure is required, and could improve the metric properties of the four original scales. Nevertheless, content validity in terms of the theoretically expected attributes of PC would be reduced as this scale retains fewer than half of the original items. Moreover, if this overall scale is computed it would not include the items relating to the comprehensiveness of primary care services available and would not apply to individuals who did not visit a specialist during the previous year, as they are not required to respond to the questions on coordination.
Using the original scale structure as the 'gold standard', reliability as measured using Cronbach's alpha was suboptimal for some of the individual scales. However, these results derive from a very small subset of the original scales, which makes it difficult to achieve a desired level of internal consistency reliability. When we calculated reliability based on the new factor structure, the alpha coefficients improved considerably suggesting that it would be more appropriate to base scoring of this reduced version on the factorial structure observed in the present study.
The construct validity analysis confirmed the expected pattern of association between experiences with PC and the item assessing satisfaction with health services. Due to a lack of instruments in Spanish measuring similar concepts, we were unable to assess convergent validity. This is a point which should be addressed in the future if such instruments become available. Another aspect which was not assessed was test-retest reliability, which would provide evidence of the degree of random error between two measures from the same individual.
The items used in this very short version of the PCAT were adapted for use in Spain following internationally recommended principles and methods for cross-cultural adaptation to maintain semantic equivalence between language versions. This makes it possible to compare results obtained here with those from the US or other countries in which the same subset of PCAT items has been applied.
The PCAT set of instruments allows for a variety of possibilities in health services research aimed at informing policy makers about the adequacy of different attributes of primary care. The Catalan Health Survey was the first large scale study including a selection of items from PCAT to assess the quality of Primary Health Care. This selection of items for a population health survey is particularly important because it provides a population perspective and can be useful to study inequalities in experiences with fundamental attributes of primary care. This very short questionnaire could be useful as a measure of primary care experiences if space is at a premium, as is often the case in population health surveys. Based on preliminary evidence of construct validity and reliability provides a summary score of judgments about experiences with primary care overall. It also can report scores on five domains of the theoretical multidimensional model on the base of their content validity and acceptable metric properties, even though lower reliability in some domains was observed and must be took into account. However, the loss of content stemming from the use of a subset of items, argues for the use of the longer versions of the PCAT to capture all the expected domains of primary care, when possible.
World Health Report 2008: Primary health care: now more than ever. 2008, Geneva: World Health Organization
Plaza Tesías A, Zara Yahni X, Guarga Rojas A, Farrés Quesada J: Resultado de la aplicación del benchmarking en los equipos de atención primaria de Barcelona. Aten Primaria. 2005, 35: 122-9. 10.1157/13071936.
Villalbí JR, Pasarín MI, Montaner I, Cabezas C, Starfield By Grupo de Trabajo sobre Evaluación en la Atención Primaria de Salud de Barcelona: Evaluación de la atención primaria de salud. Aten Primaria. 2003, 31: 382-5.
Sibthorpe B: A proposed conceptual framework for performance assessment in primary care: a tool for policy and practice. 2004, Canberra: Australian primary health care research institute, Accessed May 7, 2010, [http://www.anu.edu.au/aphcri/Publications/conceptual_framework.pdf]
Hogg W, Rowan M, Russell G, Geneau R, Muldoon L: Framework for primary care organizations: the importance of a structural domain. Int J Qual Health Care. 2008, 20 (5): 308-313.
Donabedian A: Aspects of Medical Care Administration: Specifying Requirements for Health Care. 1973, Cambridge: Harvard University Press, 419-473.
Campbell SM, Roland MO, Buetow SA: Defining quality of care. Soc Sci Med. 2000, 51 (11): 1611-1625. 10.1016/S0277-9536(00)00057-5.
Donaldson MS, Yordy KD, Vanselow NA, (eds): Institute of Medicine: Defining Primary Care: an interim report. 1994, Washington, DC: National Academy Press
Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA: Measuring consumer experiences with primary care. Pediatrics. 2000, 105: 998-1003.
Wong SY, Kung K, Griffiths SM, Carthy T, Wong MC, Lo SV, Chung VC, Goggins WB, Starfield B: Comparison of primary care experiences among adults in general outpatient clinics and private general practice clinics in Hong Kong. BMC Public Health. 2010, 10 (1): 397-10.1186/1471-2458-10-397.
Tsai J, Shi L, Yu WL, Lebrun LA: Usual source of care and the quality of medical care experiences: a cross-sectional survey of patients from a Taiwanese community. Med Care. 2010, 48 (7): 628-34. 10.1097/MLR.0b013e3181dbdf76.
Russell GM, Dahrouge S, Hogg W, Geneau R, Muldoon L, Tuna M: Managing chronic disease in ontario primary care: the impact of organizational factors. Ann Fam Med. 2009, 7 (4): 309-18. 10.1370/afm.982.
Figueiredo TM, Villa TC, Scatena LM, Cardozo Gonzales RI, Ruffino-Netto A, Nogueira Jde A, Oliveira AR, Almeida SA: Performance of primary healthcare services in tuberculosis control. Rev Saude Publica. 2009, 43 (5): 825-31. 10.1590/S0034-89102009005000054.
Starfield B: Primary care: balancing health needs, services and technology. 1998, New York. Oxford University Press
Wagstaff A, van Doorslaer E, van der Burg H, Calonge S, Christiansen T, Citoni G, Gerdtham UG, Gerfin M, Gross L, Häkinnen U, Johnson P, John J, Klavus J, Lachaud C, Lauritsen J, Leu R, Nolan B, Perán E, Pereira J, Propper C, Puffer F, Rochaix L, Rodríguez M, Schellhorn M, Winkelhake O: Equity in the finance of health care: some further international comparisons. J Health Econ. 1999, 18 (3): 263-90. 10.1016/S0167-6296(98)00044-7.
Health Survey 2006 (Enquesta de salut 2006): Proxy-respondent questionnaire for children (qüestionari informador indirecte per a menors). [http://www.gencat.cat/salut/depsalut/html/ca/plasalut/doc32868.html]
Starfield B, Shi L: PCAT Manual. 2008, Johns Hopkins University, Baltimore, [http://www.jhsph.edu/pcpc/pca_tools.html]
Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, Erikson P, ISPOR Task Force for Translation and Cultural Adaptation: Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Health. 2005, 8: 94-104. 10.1111/j.1524-4733.2005.04054.x.
Pasarín MI, Berra S, Rajmil L, Solans M, Borrell C, Starfield B: An instrument to evaluate primary health care from the population perspective. Aten Primaria. 2007, 39 (8): 395-401. 10.1157/13108612.
Rochas KB, Rodríguez-Sanz M, Pasarín M, Berra S, Gotsens Miquel M, Borrell C: Assessment of primary health care attributes in health surveys: a population perspective. Eur J Public Health. 2011
Blendon RJ, Donelan K, Jovell AJ, Pellisé L, Costas-Lombardía E: Spain's citizen assess their health care system. Health Affairs. 1991, 10 (6): 216-28.
Idescat. Statistical Institute of Catalonia: Statistical Yearbook of Catalonia. Structure of the by sex and age groups. 2006, [http://www.idescat.cat/pub/?id=aec&n=253&lang=en]
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/11/285/prepub
This study was partially funded by the Fondo de Investigaciones Sanitarias, Spanish Ministry of Science and Innovation (contract PI08/90698). Authors are grateful to Carlos García Forero for his assistance with the data analysis, and Michael Herdman for the review of the manuscript.
The authors declare that they have no competing interests.
SB and MIP drew up the initial project design; MRS, LR, CB, BS participated in the study design and contributed to the data analysis plan; KBR carried out part of the statistical analysis and the first draft of the results; SB drafted the manuscript; all authors revised it critically, made contributions to the interpretation of data, and gave final approval of the version to be published.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.