Properties of a short questionnaire for assessing Primary Care experiences for children in a population survey
© Berra et al; licensee BioMed Central Ltd. 2011
Received: 9 July 2010
Accepted: 9 May 2011
Published: 9 May 2011
The Primary Care Assessment Tool (PCAT) is an interesting set of tools for primary care research. A very short version could inform policy makers about consumer experiences with primary care (PC) through health surveys. This work aimed to investigate the validity and reliability of a selection of items from the child short edition (CS) of the PCAT.
A 24 item questionnaire permitted the identification of a regular source of care and the assessment of the key attributes of first contact, ongoing care over time, coordination, services available and services received (comprehensiveness), and cultural competence. Structural validity, reliability, and construct validity were assessed using responses from 2,200 parents of a representative sample of the population aged 0 to 14 years in Catalonia (Spain) who participated in the 2006 Health Survey. Structural validity was analyzed using exploratory and confirmatory factor analyses and reliability was assessed using Cronbach's alpha. Construct validity was assessed using linear regression analysis between PC experience scores and a measure of overall user satisfaction with healthcare services.
A total of 2,095 (95.2%) parents provided useable responses on PC. After Confirmatory Factor Analysis (CFA), the best fitting model was a 5-factor model in which the original dimensions of first contact and ongoing care were collapsed into one. The CFA also showed a second order factor onto which all domains except services available loaded (root mean square error of approximation = 0.000; comparative fit index = 1.00). Cronbach's alpha values for one of the original scales (first-contact) was poor (alpha < 0.50), but improved using the modified factor structure (alpha > 0.70). Scores on the scales were correlated with satisfaction with healthcare services (p < 0.01), thereby providing some preliminary evidence of construct validity.
This very short questionnaire obtained from the PCAT-CE yields information about five attributes of PC and a summary score. It has shown evidence of validity and reliability for judgments about experiences with primary care overall. If space on surveys is at a premium, the instrument could be useful as a measure of PC experiences.
The importance of primary care (PC) in making health care delivery more efficient and in tackling health inequalities has been discussed at a political level worldwide for at least the last three decades . Several countries have reformed their health system with a particular emphasis on strengthening primary care. Public health researchers have made considerable efforts to assess the quality of primary care and the extent of improvements. Interest in a broader theoretical model of PC has also grown, and it has become clear that there is a need for research tools to measure important attributes of primary care from the user's point of view [2, 3].
Theoretical models of primary care have stressed its structural and organizational aspects [4, 5] and other attributes essential to primary contact for users and their families, focusing on the user perspective [6, 7]. One of the most widely acknowledged definitions of the quality of PC services is "the provision of accessible and integrated services addressing a large majority of health care needs, developing a sustained relationship with patients, and practicing in the context of family and community" . The Primary Care Assessment Tools (PCAT), developed in the United States (US), addresses these issues .
The PCAT is a set of questionnaires consisting of consumer-client surveys, facility surveys, and provider surveys. Their development provided a set of tools for PC evaluation that have become increasingly widely used in several countries [10–13]. The different versions include items designed to collect information on four core and three ancillary domains of PC. The core attributes are : first contact, i.e. assessment of PC's role as the entry point to the health care system except in emergency situations; ongoing care, i.e. maintenance of person-focused care over time; comprehensiveness, which deals with PC's ability to make available and provide a wide range of services, including but not limited to preventive care, in response to prevalent health needs; and coordination, i.e. to support and integrate the care of problems addressed elsewhere, either by practitioners or medical records. PCAT also addresses the extent to which PC focuses on the health of individuals within the context of family (family orientation), is able to tackle community health problems (community orientation), and can deal with the need to establish relationships with people from different social groups (cultural competence).
Spain initiated substantial health care reforms in 1986, by legislating for universal tax-financed services, increasing the proportion of total health expenditure which is publicly financed,  and making the strengthening of primary care teams a central part of its strategy in primary care. The state-run health service provides free access to services, but approximately 10% of adult people and one third of the infant population pay for additional private healthcare coverage. Autonomous Communities have full autonomy in the health-care sector in Spain. In Catalonia, in the north-east of Spain, health interview surveys have been conducted periodically since 1983 (Barcelona city) and 1994 (Catalonia) to evaluate the population's health status and its determinants, and to assess the performance of healthcare services. In the 2006 edition of the Catalan Health Interview Survey, the intention to assess Primary Health Care led to interest in implementing the PCAT. Since the expanded version of the PCAT for children (PCAT-CE) or the short version (PCAT-CS) could not be included due to limitations in the length of the survey, several items from the PCAT-CS were selected to provide information on primary care. Given that use of this subset of items represented a substantial modification to the original instrument, it was considered necessary to re-assess its psychometric characteristics. The research question addressed in this paper is whether the selected items provide a valid and reliable measure of parent experiences with PC.
Sample and setting
The 2006 Catalan Health Interview Survey was a cross-sectional study carried out in a representative sample of non-institutionalized residents of Catalonia. Sample selection used a multi-stage design. First, municipalities were selected according to number of inhabitants and, second, individuals were selected based on age and sex distribution from the population registry of the Statistical Institute of Catalonia (IDESCAT). The sample size for the population under 15 years was established at 2,200 infants and children. Main cases and five replacements were initially selected. The sample size was reached with 65% of main cases and 22% of first replacement cases; the remainders were mostly second replacement cases. Reasons for replacement were because parents refused the interview or were not found at the available address. The survey questionnaire was administered during home-based interviews with a proxy respondent, preferably the child's usual caregiver.
The instrument to assess PC in Spain
The original child edition of the short, consumer version of the PCAT contains 44 items (PCAT-CS) distributed over 7 domains . Five initial questions are used to identify the source of Primary Care. First contact accessibility and utilization (first contact domain), longitudinal interpersonal relationships (ongoing care domain), coordination of services (coordination domain), comprehensiveness of services available and received (comprehensiveness domain) are assessed through questions on how confidant the respondent is about their provision by the primary care source. Three ancillary domains (family centeredness, community orientation, and cultural competence) are also assessed.
To include the PCAT in the 2006 Catalan Health Interview Survey questionnaire for 0 to 14 year olds, it was necessary to select a subset of items due to limitations on space. Based on the PCAT-CS, the local working group together with one of the authors of the original version agreed on a very short version that was considered to best represent the thrust of the original questionnaire. This compromise led to the exclusion of the Community and Family Orientation domains, and the selection of a minimum of two items from the remaining domains based on their interest in covering a broader range of aspects of primary care performance. The final version included twenty-four items adapted to the Spanish context, since the Health System is homogeneous in all the Autonomous Communities in Spain. Two initial questions ask if children have a regular source of primary care or, if not, about the last general practitioner or pediatrician that they visited. Three questions characterize the regular source of care. The remaining 19 items cover five of the seven domains in the original questionnaire, including first contact (4 items, mostly addressing accessibility), ongoing care (3 items), coordination of services (a filter plus 2 items), comprehensiveness (4 items in the services available subdomain, and (3 items of the services received subdomain, one of which is only for adolescents), and cultural competence (2 items). Items and other text were translated into Catalan and Spanish, the main official languages spoken in Catalonia, following an internationally recommended process of cross-cultural adaptation to achieve semantic equivalence with the original version . Further details of the selection of items and the cross-cultural adaptation have been published elsewhere . The current analysis focuses only on the 17 items actually used to assess PC attributes across the full age-range for paediatric care. The 5 items used to identify and characterise the source of care were therefore excluded, together with the filter question before the coordination domain, and one item which only applies to adolescents.
All items covering the PC domains are answered on a 4-point Likert-type scale (1 = definitely not; 2 = probably not; 3 = probably yes; and 4 = definitely yes). Additional response options included "don't know" or "can't remember" . The score for each domain is computed as the mean value for all items in that domain, and can range between 1 and 4.
The Catalan Health Interview Survey is an observational health survey conducted by the Department of Health of Catalonia as part of routine governmental statistics gathering. The survey complies with all relevant national legislation on the protection and processing of personal data. Data are not openly available but can be obtained on request from the Catalan Department of Health, and the entire questionnaire can be accessed in its website .
Demographic and health characteristics were described for the sub-sample of the Catalan Health Interview Survey used in the present analysis. As those who did not identify a regular source of care were excluded from the remainder of the analysis, they were compared with those included using chi-square to test for differences on demographic and health status variables.
Several factor analyses were carried out to evaluate the structural validity of this modified version of the PCAT. Factor analysis is a multivariate technique that allows the implementation of a statistical model that reproduces the item variance-covariance matrix and provides statistical information on model quality. In the first instance, we used exploratory factor analysis (EFA) to extract an increasing number of first order factors and account for observed variability without further restrictions on the model. Factors were extracted using diagonally weighted least squares (DWLS) on the item polychoric correlation matrix. Oblimin rotation was applied to allow for correlation between factors. We hypothesized that items would load onto domains in the same way as in the original questionnaire, thereby indicating the appropriateness of the underlying conceptual model for the Catalan population. Preliminary results from this initial factor analysis were further tested using Confirmatory Factor Analysis (CFA). In this case, certain restrictions were imposed on factor structure, and the presence of second order factors and the structural validity of a global score encompassing all domains were also tested. The unweighted least squares estimation method on the item polychoric correlation matrix was used to obtain p-values and standard errors robust for non-normal distributions. This method respects the ordered categorical nature of the items while providing robust estimates of factor loading. The model's overall goodness of fit was assessed using the root mean square error of approximation (RMSEA), and two incremental fit indices, the comparative fit index (CFI), and the Tucker Lewis-Index (TLI). A RMSEA under 0.06 was considered to indicate an excellent fit between the specified model and the data, whereas values of CFI and TLI were required to be over 0.95. The analyses were repeated separately for infants (0 to 5 years), children (6 to 11 years) and adolescents (12 to 14 years) to take into account possible structural variance across age groups.
Models with the best results in the EFA were studied in more depth using confirmatory factor analysis. Pearson correlation coefficients were used to determine the degree of correlation between domains; some degree of correlation between factors was expected.
The descriptive and reliability analysis was conducted based on both the questionnaire structure as administered and the new structure stemming from factor analysis. Score distributions for the 17 items of the PCAT domains was carried out by calculating the mean and standard deviation, the proportion of missing responses, the observed score range, and floor and ceiling effects (i.e. the proportion of cases with the worst and best possible score, respectively). The percentage of items which correlated higher with their hypothesized domain than with the other domains obtained in the EFA (scaling success rates) was calculated. Homogeneity between items in each factor derived from the factor analysis was assessed using Cronbach's alpha.
General satisfaction with services was used to assess construct validity. Satisfaction was analyzed by an item in the main survey questionnaire with four-point Likert-type response scale where 1 = very satisfied, and 4 = very dissatisfied . Validity was assessed by using individual linear regression models with each factor of the PCAT to predict the dissatisfaction score. We expected to find a moderate association between user satisfaction scores and scores derived for each factor of the PC experience scales. Associations were expected to be negative because satisfaction is scored inversely (1 = satisfied) to the PC experience scales (1 = negative).
SPSS 15.0 was used for the descriptive analysis and Mplus version 5.2 was used for the factorial and regression analyses.
Demographic and heath characteristics of the sample.
Identifying an usual source of care
0 - 5 years
6 - 11 years
12 - 14 years
Educational level. Maximum for father or mother.
Less than primary
Double (private and public)
Perceived child's health
Declared chronic conditions a
Exploratory factor analysis for the PCAT-CE, very short version.
Domains and items
FIRST CONTACT ACCESSIBILITY AND ONGOING CARE
When your child has a new health problem, do you go to your PCP before going somewhere else?
When your PCP is open and your child gets sick, would someone from there see him/her the same day?
When your PCP is open, can you get advice quickly over the phone if you need it?
When your PCP is closed, is there a phone number you can call when your child gets sick?
When you take your child to your PCP's, is s/he taken care of by the same doctor or nurse each time?
If you have a question, can you call and talk to the doctor or nurse who knows your child best?
Does your PCP know what problems are most important to you and your family?
Did your PCP write down any information for the specialist about the reason for the visit?
After your child went to the specialist or special service, did your PCP talk with you and your child about what happened at the visit?
COMPREHENSIVENESS (SERVICES AVAILABLE)
Family planning or birth control methods
Counseling for behavior or mental health problems
Sewing up a cut that needs stitches
COMPREHENSIVENESS (SERVICES PROVIDED)
Ways to keep your children healthy, such as nutritional foods or getting enough sleep
Ways to handle problems with your child's behavior
Would you recommend your child's PCP to a friend or relative who has a child?
Would you recommend your child's PCP to someone who does not speak English well?
Score distribution and reliability of the PCAT domains - original and modified scaling.
Num. of items
% Scaling succes
Cronbach's alpha (SE)
First Contact Accessibility
Comprehensiveness (Services Available)
Comprehensiveness (Services Provided)
First Contact Accessibility & Ongoing Care
Experience with PC (second order factor) b
First Contact Accessibility
Comprehensiveness (Services Available)
Comprehensiveness (Services Provided)
First Contact & Ongoing Care
Experience with PC (second order factor)
This study analyzed the structure, construct validity and reliability of a very short version of the PCAT for parents of children aged 0 - 14 years. This instrument was derived from the original short version developed by researchers in the US (PCAT-CS), and its seventeen items measure several attributes of PC, based on the questionnaire's original conceptual model, with acceptable reliability and validity.
An important finding from the study included the fact that items about ongoing care correlated highly with those measuring first contact accessibility, to the extent that they all loaded onto a single factor. The original PCAT 'ongoing care' domain contained two types of items: those concerning identification with a particular provider and those concerning the nature of the relationship. In the very abridged version used in this study, 2 of the 3 items from the subdomain of interpersonal relationships were associated, making them similar in concept to the items on access in the first-contact subdomain. There is thus a difference with the original instrument, since the smaller number of items and the traits captured by them cannot collect equal information addressing the nature of the relationship, i.e., person focused care over time. Apart from this limitation, the analysis of structural validity indicated that the items selected adequately represent the original conceptual multidimensional model because data fitted well to the remaining 4 domains of coordination, services available, services provided, and cultural competence.
The bifactorial model was also confirmed as an alternative measure of two concepts, with good statistical fit. 'Services available' (a list of characteristics of the source of care that might be needed at some time) is different from services received and, hence, not strictly an 'experiences' measure. Moreover, the second order factor resulting from the confirmatory factor and its reliability analysis underlined the possibility of gathering the information on a unique scale. This option might be useful when a single global measure is required, and could improve the metric properties of the four original scales. Nevertheless, content validity in terms of the theoretically expected attributes of PC would be reduced as this scale retains fewer than half of the original items. Moreover, if this overall scale is computed it would not include the items relating to the comprehensiveness of primary care services available and would not apply to individuals who did not visit a specialist during the previous year, as they are not required to respond to the questions on coordination.
Using the original scale structure as the 'gold standard', reliability as measured using Cronbach's alpha was suboptimal for some of the individual scales. However, these results derive from a very small subset of the original scales, which makes it difficult to achieve a desired level of internal consistency reliability. When we calculated reliability based on the new factor structure, the alpha coefficients improved considerably suggesting that it would be more appropriate to base scoring of this reduced version on the factorial structure observed in the present study.
The construct validity analysis confirmed the expected pattern of association between experiences with PC and the item assessing satisfaction with health services. Due to a lack of instruments in Spanish measuring similar concepts, we were unable to assess convergent validity. This is a point which should be addressed in the future if such instruments become available. Another aspect which was not assessed was test-retest reliability, which would provide evidence of the degree of random error between two measures from the same individual.
The items used in this very short version of the PCAT were adapted for use in Spain following internationally recommended principles and methods for cross-cultural adaptation to maintain semantic equivalence between language versions. This makes it possible to compare results obtained here with those from the US or other countries in which the same subset of PCAT items has been applied.
The PCAT set of instruments allows for a variety of possibilities in health services research aimed at informing policy makers about the adequacy of different attributes of primary care. The Catalan Health Survey was the first large scale study including a selection of items from PCAT to assess the quality of Primary Health Care. This selection of items for a population health survey is particularly important because it provides a population perspective and can be useful to study inequalities in experiences with fundamental attributes of primary care. This very short questionnaire could be useful as a measure of primary care experiences if space is at a premium, as is often the case in population health surveys. Based on preliminary evidence of construct validity and reliability provides a summary score of judgments about experiences with primary care overall. It also can report scores on five domains of the theoretical multidimensional model on the base of their content validity and acceptable metric properties, even though lower reliability in some domains was observed and must be took into account. However, the loss of content stemming from the use of a subset of items, argues for the use of the longer versions of the PCAT to capture all the expected domains of primary care, when possible.
This study was partially funded by the Fondo de Investigaciones Sanitarias, Spanish Ministry of Science and Innovation (contract PI08/90698). Authors are grateful to Carlos García Forero for his assistance with the data analysis, and Michael Herdman for the review of the manuscript.
- World Health Report 2008: Primary health care: now more than ever. 2008, Geneva: World Health OrganizationGoogle Scholar
- Plaza Tesías A, Zara Yahni X, Guarga Rojas A, Farrés Quesada J: Resultado de la aplicación del benchmarking en los equipos de atención primaria de Barcelona. Aten Primaria. 2005, 35: 122-9. 10.1157/13071936.View ArticlePubMedGoogle Scholar
- Villalbí JR, Pasarín MI, Montaner I, Cabezas C, Starfield By Grupo de Trabajo sobre Evaluación en la Atención Primaria de Salud de Barcelona: Evaluación de la atención primaria de salud. Aten Primaria. 2003, 31: 382-5.View ArticlePubMedGoogle Scholar
- Sibthorpe B: A proposed conceptual framework for performance assessment in primary care: a tool for policy and practice. 2004, Canberra: Australian primary health care research institute, Accessed May 7, 2010, [http://www.anu.edu.au/aphcri/Publications/conceptual_framework.pdf]Google Scholar
- Hogg W, Rowan M, Russell G, Geneau R, Muldoon L: Framework for primary care organizations: the importance of a structural domain. Int J Qual Health Care. 2008, 20 (5): 308-313.View ArticlePubMedGoogle Scholar
- Donabedian A: Aspects of Medical Care Administration: Specifying Requirements for Health Care. 1973, Cambridge: Harvard University Press, 419-473.Google Scholar
- Campbell SM, Roland MO, Buetow SA: Defining quality of care. Soc Sci Med. 2000, 51 (11): 1611-1625. 10.1016/S0277-9536(00)00057-5.View ArticlePubMedGoogle Scholar
- Donaldson MS, Yordy KD, Vanselow NA, (eds): Institute of Medicine: Defining Primary Care: an interim report. 1994, Washington, DC: National Academy Press
- Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA: Measuring consumer experiences with primary care. Pediatrics. 2000, 105: 998-1003.PubMedGoogle Scholar
- Wong SY, Kung K, Griffiths SM, Carthy T, Wong MC, Lo SV, Chung VC, Goggins WB, Starfield B: Comparison of primary care experiences among adults in general outpatient clinics and private general practice clinics in Hong Kong. BMC Public Health. 2010, 10 (1): 397-10.1186/1471-2458-10-397.View ArticlePubMedPubMed CentralGoogle Scholar
- Tsai J, Shi L, Yu WL, Lebrun LA: Usual source of care and the quality of medical care experiences: a cross-sectional survey of patients from a Taiwanese community. Med Care. 2010, 48 (7): 628-34. 10.1097/MLR.0b013e3181dbdf76.View ArticlePubMedGoogle Scholar
- Russell GM, Dahrouge S, Hogg W, Geneau R, Muldoon L, Tuna M: Managing chronic disease in ontario primary care: the impact of organizational factors. Ann Fam Med. 2009, 7 (4): 309-18. 10.1370/afm.982.View ArticlePubMedPubMed CentralGoogle Scholar
- Figueiredo TM, Villa TC, Scatena LM, Cardozo Gonzales RI, Ruffino-Netto A, Nogueira Jde A, Oliveira AR, Almeida SA: Performance of primary healthcare services in tuberculosis control. Rev Saude Publica. 2009, 43 (5): 825-31. 10.1590/S0034-89102009005000054.View ArticlePubMedGoogle Scholar
- Starfield B: Primary care: balancing health needs, services and technology. 1998, New York. Oxford University PressGoogle Scholar
- Wagstaff A, van Doorslaer E, van der Burg H, Calonge S, Christiansen T, Citoni G, Gerdtham UG, Gerfin M, Gross L, Häkinnen U, Johnson P, John J, Klavus J, Lachaud C, Lauritsen J, Leu R, Nolan B, Perán E, Pereira J, Propper C, Puffer F, Rochaix L, Rodríguez M, Schellhorn M, Winkelhake O: Equity in the finance of health care: some further international comparisons. J Health Econ. 1999, 18 (3): 263-90. 10.1016/S0167-6296(98)00044-7.View ArticlePubMedGoogle Scholar
- Health Survey 2006 (Enquesta de salut 2006): Proxy-respondent questionnaire for children (qüestionari informador indirecte per a menors). [http://www.gencat.cat/salut/depsalut/html/ca/plasalut/doc32868.html]
- Starfield B, Shi L: PCAT Manual. 2008, Johns Hopkins University, Baltimore, [http://www.jhsph.edu/pcpc/pca_tools.html]Google Scholar
- Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, Erikson P, ISPOR Task Force for Translation and Cultural Adaptation: Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Health. 2005, 8: 94-104. 10.1111/j.1524-4733.2005.04054.x.View ArticlePubMedGoogle Scholar
- Pasarín MI, Berra S, Rajmil L, Solans M, Borrell C, Starfield B: An instrument to evaluate primary health care from the population perspective. Aten Primaria. 2007, 39 (8): 395-401. 10.1157/13108612.View ArticlePubMedGoogle Scholar
- Rochas KB, Rodríguez-Sanz M, Pasarín M, Berra S, Gotsens Miquel M, Borrell C: Assessment of primary health care attributes in health surveys: a population perspective. Eur J Public Health. 2011Google Scholar
- Blendon RJ, Donelan K, Jovell AJ, Pellisé L, Costas-Lombardía E: Spain's citizen assess their health care system. Health Affairs. 1991, 10 (6): 216-28.View ArticlePubMedGoogle Scholar
- Idescat. Statistical Institute of Catalonia: Statistical Yearbook of Catalonia. Structure of the by sex and age groups. 2006, [http://www.idescat.cat/pub/?id=aec&n=253&lang=en]Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/11/285/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.