Quality in Primary Care Open Access

  • ISSN: 1479-1064
  • Journal h-index: 26
  • Journal CiteScore: 7.78
  • Journal Impact Factor: 4.76
  • Average acceptance to publication time (5-7 days)
  • Average article processing time (30-45 days) Less than 5 volumes 30 days
    8 - 9 volumes 40 days
    10 and more volumes 45 days
Reach us +32 25889658

Research Paper - (2013) Volume 21, Issue 2

Measures of quality, costs and equity in primary health care: instruments developed to analyse and compare primary health care in 35 countries

Willemijn LA Scha¨ fer*

Wienke GW Boerma

NIVEL, Netherlands Institute for Health Services Research, Utrecht, The Netherlands

Dionne S Kringos

NIVEL, Netherlands Institute for Health Services Research, Utrecht, The Netherlands and Department of Social Medicine, Academic Medical Centre (AMC), University of Amsterdam, The Netherlands

Evelyne De Ryck

Department of General Practice and Primary Health Care, Ghent University, Belgium

Stefan Greß and Stephanie Heinemann

Hochschule Fulda – University of Applied Sciences, Fulda, Germany

Anna Maria Murante

Laboratorio MeS, Istituto di Management, Scuola Superiore Sant’Anna, Italy

Danica Rotar-Pavlic

ULMF – University of Ljubljana, Slovenia

Franc¸ ois G Schellevis

NIVEL, Netherlands Institute for Health Services Research, Utrecht, The Netherlands and Department of General Practice and Elderly Care Medicine, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands

Chiara Seghieri

Laboratorio MeS, Istituto di Management, Scuola Superiore Sant’Anna, Italy

Michael J Van den Berg

RIVM – National Institute for Public Health and the Environment, Bilthoven, The Netherlands and Tranzo, Tilburg University, The Netherlands

Gert P Westert

Radboud University Nijmegen Medical Centre, The Netherlands

Sara Willems

Department of General Practice and Primary Health Care, Ghent University, Belgium

Peter P Groenewegen

NIVEL, Netherlands Institute for Health Services Research, Utrecht, The Netherlands and Department of Sociology and Department of Human Geography, Utrecht University, The Netherlands

*Corresponding Author:
Willemijn LA Scha¨fer
NIVEL, Netherlands Institute for Health Services Research
Utrecht, The Netherlands
Email: [email protected]

Received date: 24 October 2012; Accepted date: 25 January 2013

Visit for more related articles at Quality in Primary Care


BackgroundThe Quality and Costs of Primary Care in Europe (QUALICOPC) study aims to analyse and compare how primary health care systems in 35 countries perform in terms of quality, costs and equity. This article answers the question ‘How can the organisation and delivery of primary health care and its outcomes be measured through surveys of general practitioners (GPs) and patients?’ It will also deal with the process of pooling questions and the subsequent development and application of exclusion criteria to arrive at a set of appropriate questions for a broad international comparative study. MethodsThe evelopment of the questionnaires consisted of four phases: a search for existing validated questionnaires, the classification and selection of relevant questions, shortening of the questionnaires in three consensus rounds and the pilot survey. Consensus was reached on the basis of exclusion criteria (e.g. the pplicability for international comparison). Based on the pilot survey, comprehensibility increased and the number of questions was further restricted, as the questionnaires were too long. ResultsFour uestionnaires were developed: one for GPs, one for patients about their experiences with their GP, another for patients about what they consider important, and a practice questionnaire. The GP uestionnaire mainly focused on the structural aspects (e.g. economic conditions) and care processes (e.g. comprehensiveness of services of primary care). The patient experiences questionnaire focused on the care processes and outcomes (e.g. how do patients experience access to care?). The questionnaire about what patients consider important was complementary to the experiences questionnaire, as it enabled weighing the answers from the latter. Finally, the practice questionnaire included questions on practice characteristics. DiscussionThe QUALICOPC researchers have developed four questionnaires to characterise the organisation and delivery of primary health care and to compare and analyse the outcomes. Data collected with these instruments will allow us not only to show in detail the variation in process and outcomes of primary health care, but also to explain the differences from features of the (primary) health care system.


health care surveys, health services research, outcome assessment (health care), primary health care


Many European countries share the goal of initiating or sustaining strong primary health care systems. As a result, there is a demand for benchmarking information and a growing tendency to learn from foreign experiences. Evidence on the outcomes of primary health care in European countries is, however, still incomplete.[1] Variation in the organisation of primary health care in Europe enables analyses of the relationship between primary care organisation and outcomes. Decision makers may benefit from information about arrangements of primary health care which are more likely to produce better outcomes.[2]

In 2010, the three-year Quality and Costs of Primary Care in Europe (QUALICOPC) study started. This study aims to compare and analyse how the primary health care systems of 35 countries perform in terms of quality, costs and equity. The results of this study will contribute to evidence on the benefits of strong primary health care and on the performance of health care systems in general. The European countries include 27EUcountries, Iceland,Norway,Turkey, Switzerland and Macedonia. Outside Europe, Australia, Canada and New Zealand have joined the study. For this study, data will be gathered by means of surveys among general practitioners (GPs) and their patients.[1] National characteristics of the organisation of primary health care will be derived from other sources, such as the Primary Health Care Activity Monitor (PHAMEU) database.[3] GPs were chosen as survey subjects, because they can be seen as the main providers of primary health care. However, the project aims to provide insight into not only GP care, but also primary health care as a whole. Fieldworkers, who will visit GP practices to recruit patients and assist them, if necessary, with filling in the questionnaire, will also fill in a practice questionnaire. The data from GPs, patients and fieldworkerswill be linked to each other. For more information on the QUALICOPC study, see Box 1.[1]

Primary health care can be characterised as the first level of access to care and is provided near patients’ homes. Primary health care includes curative and rehabilitative care, preventive care and health education. [4,5] A recent literature review on primary health care6 distinguishes three levels of care, namely the structure, process and outcome of care. Within these levels, 10 core dimensions to measure primary health care were identified (see Table 1).

The QUALICOPC study aims to comprehensively evaluate the breadth of primary health care by gathering data on all these dimensions.

The analyses will focus on the following overarching themes: quality of the process of care (including the dimensions of access, continuity, coordination and comprehensiveness of primary health care services), experiences of patients (as an indicator of the dimension quality of primary health care), costs of primary health care (as a part of the dimension efficiency of care), equity (related to the dimension access and the quality of primary health care), avoidable hospitalisation (as an indicator of the dimension quality of primary health care). A sixth synthesising theme will be the identification of ‘good practices’ of primary health care provision (related to all dimensions to measure primary health care).[1]



To collect data related to these six themes, new questionnaires had to be developed. Many previous studies have used questionnaires for primary health care physicians and patients. In the past, comprehensive primary care studies have been performed, for example by Barbara Starfield[7] and the Commonwealth Fund,[8] but only a limited number of European countries were included. Furthermore, many studies that have used questionnaires from GPs and patients had a focus on specific subjects or themes rather than a multidimensional approach. This study aims to unravel the processes and contributions of primary health care to its outcomes in terms of quality, costs and equity. The questions in the questionnaire should not only cover all themes but also be suitable for use in international surveys, which means that differences in the health care context between countries need to be taken into account.

This article describes the background to and development of the questionnaires for the QUALICOPC study. It addresses the question ‘How can the quality, costs and equity of a primary health care system be measured?’ Furthermore, criteria used for inclusion or exclusion of questions are presented, as well as an overview of the resulting questions that can be used for international comparative research on primary health care.


Four questionnaires were developed: one for GPs, one for patients about their experiences with their GP, another for patients about their values regarding primary care (i.e. what they consider important), and finally one about the practice. Because the project aims to provide insight into GP care as a whole, the GP questionnaire should also include questions beyond the scope of the tasks of the GP. The questionnaire about what patients find important is added to weigh against their experiences. Development of the questionnaires consisted of four phases: a search for existing questionnaires, the classification and selection of relevant questions (including formulation of inclusion and exclusion criteria), shortening of the questionnaires and the pilot survey. An overview of the development process is presented in Figure 1.

Phase 1: bibliometric search

In the first phase, existing questionnaires, published between 1990 and 2010 and with an abstract written in English, were searched for in the bibliographic databases PubMed and Embase. The search aimed to identify validated questionnaires for primary health care physicians and patients, suitable for international comparisons. Search terms were derived from the 10 dimensions formeasuring primary health care (Table 1). In addition, attention was paid to identifying questionnaires on avoidable hospitalisation, which is not explicitly covered in the dimensions, and on equity, which has received relatively little attention in international comparative primary health care research.[2,6]

Phase 2: classification, selection, rephrasing and new questions

In the second phase, questions from the included questionnaires were classified according to the 10 dimensions. Next, the researchers selected questions that contribute to answering the main research questions of the QUALICOPC study. Questions were rephrased to fit the study approach and aim. Furthermore, new questions were formulated for gaps that were identified. The identified questions were divided between the provisional list of questions for the GP questionnaire and the Patient Experiences and Patient Values questionnaires.

Phase 3: consensus rounds

Next, in three consensus rounds, the researchers evaluated the questionnaires and selected the questions for inclusion. Each of the questions was discussed for its relevance to the purpose of this study and the exclusion criteria in order to further increase the suitability of the questions for the surveys. The researchers developed the following set of criteria for inclusion/exclusion:

• the question is not suitable for international comparison (e.g. not applicable in several countries)

• the question refers to a characteristic of the health care system (that can be found elsewhere, e.g. the PHAMEU database) rather than to a characteristic of an individual practice or experience of a patient

• very little variation in the answers is expected, both within and between countries

• the question is very detailed and will provide only fractional information

• answers to the questions are expected to be unreliable (e.g. due to social desirability bias)

• the question is likely to be too difficult for the respondent (e.g. it demands a high level of literacy).

In three rounds, the researchers submitted the questions to these criteria, until consensus was found. At this stage, questions were reformulated where necessary to increase comprehensibility.

Pilot survey

As a final step, a pilot survey was held with GPs and patients in Belgium, the Netherlands and Slovenia, aiming to test the practicality and applicability of the survey and the comprehensibility and appropriateness of the questions. In each country, a convenience sample of GPs (around 10) was invited to participate. GPs were asked to fill in the questionnaire, which contained an extra column to add comments and questions to the questionnaire. Furthermore, project researchers visited the general practices to recruit a random sample of patients. In each practice, four consecutive patients who agreed to fill in the questionnaire were included. This resulted in a total of 112 completed questionnaires from patients (40 in Belgium and Slovenia and 32 in the Netherlands). During the visits, researchers filled in a checklist, took notes of the proceedings and asked the patients to directly mention problems or questions which they did not understand. Based on the findings of the pilot a final consensus round was held in which the questionnaires were further shortened and questions which were found too difficult were rephrased.

During the pilot and the subsequent final consensus round, special attention was paid to the intelligibility of questions, because the changed wording of several questions could have affected their validity. Explicit cognitive testing, however, has not been part of the pilot study. For two reasons it was decided not to assess the psychometric properties of the draft questionnaires. First, questions dealing with factual circumstances or facilities are less suitable for such testing. Besides, questions copied from validated questionnaires have been tested already. For instance, the questions on services that GPs offer to their patients that are derived from the European GP Task Profile study have been tested for internal consistency and scale reliability.[9]


In this section, the results of each of the phases of the development are discussed. Next, the final outcomes, namely the questionnaires, are presented.

Phase 1: bibliometric search

Through the bibliometric search, 2783 potentially relevant studies for the GP questionnaire were identified. After careful screening, 13 relevant primary health care physician questionnaires were identified, an overview of which is presented in Box 2. For the patient questionnaire, 2213 potentially relevant sources were found, which eventually resulted in 64 relevant questionnaires (see Box 3).

Phase 2: classification, selection, rephrasing and new questions

All questions from the retrieved questionnaires were classified according to the dimensions to measure primary care. The result of this classification is presented in Table 2. As some questions were classified in more than one dimension, the total number in the figure is higher than the number of questions that emerged from the search.

For each of the dimensions, the researchers selected questions potentially relevant to this study. An example of a question which was not included in the first selection phase is about the health plans of the patients. This question is country specific and not suitable for comparison between countries. After this first phase, 138 questions for GPs and 117 for patients remained.

Phase 3: consensus rounds

During the consensus phase, the questions on the provisional were further narrowed (based on the exclusion criteria) and rephrased, where necessary. For instance, as more and more GPs work part-time, the question about the number of GP colleagues working in the same practice was further specified to include the number of full-time equivalents (FTEs) in addition to the absolute number. The number of remaining questions after each round is indicated in Figure 1.


Figure 1 Phases in the development of the questionnaires.


The pilots showed that the questionnaires were reasonably well understood and easily administered, suggesting acceptable clarity and applicability. However, both the GP and Patient Experiences questionnaires were too long, as the average time needed for completion exceeded the set limits of 30 minutes for GPs and 20 minutes for patients. Furthermore, in the GP questionnaire mistakes were identified (e.g. names of equipment were incorrect). Some questions in the patients’ questionnaire appeared too difficult.

The pilot resulted in a further reduction of the questionnaire, reformulation of several questions and the development of a short practice questionnaire about general characteristics of the practice (e.g. cleanliness of the waiting room).

GP questionnaire

The final GP questionnaire (see Appendix A – available online) contains 60 questions (25 of which have two or more subquestions). The majority of the questions have prestructured multiple choice answers. In 13 questions, GPs are also asked to fill in numerical answers (e.g. a percentage or a number of hours).




Appendix A (available online) also provides an overview of the thematic content of each of the questions and the sources used for the questions. Three questions focus on the background of the GP and four on the characteristics of the practice (e.g. the composition of the practice population). ‘Efficiency is measured by seven questions for instance on time allocation of the GP. Within the theme ‘‘workforce development’’ there are four questions, from additional professional activities of GP and disciplines working in the practice to job satisfaction. Five questions focus on ‘economic conditions’ (payment of the GP and co-payment for patients). ‘Equity in access’ is reflected in questions about restrictions in access and availability of care for uninsured patients.

To gain insight into the relationship between GPs and the broader contacts of primary care, there are 12 questions about ‘coordination and cooperation’ between GPs and other disciplines. Eleven questions about the ‘continuity of care’ provided by the GPs concentrate on disease management and on referrals and information exchange. Special attention is paid to medical record keeping.

‘Quality of care’ is measured with three questions regarding the use of guidelines and feedback from colleagues or authorities. ‘Comprehensiveness of care services’ is reflected in 12 questions, dealing with the available equipment and the GPs’ task profiles (e.g. the range of problems for which the GP is the first point of contact). Finally, nine questions covering ‘accessibility of care’ can be divided into those about physical access (distance to the practice and opening hours) and those about financial access to care services.

The European study on GP Task Profiles, carried out in 30 European countries in 1992–93, is a major source for the GP questionnaire.[10] Several questions were copied from this questionnaire. Other important sources are, for example, international surveys by the Commonwealth Fund[8,11,12] (questions about financial incentives, guideline use and medical record keeping) and Starfield’s Primary Care Assessment Tool (question about care for uninsured persons).[13]

For several topics, no examples of existing questions were found and new questions had to be formulated. These topics were involvement of GPs in disease management programmes, equity in access and patient involvement in the decision-making process.

Patient Experiences questionnaire

The Patient Experiences questionnaire, dealing with the experiences of patients with their GP (see Appendix B – available online), contains 41 multiple choice questions (10 of which have two or more subquestions). Many questions ask to what extent the patient agrees with a statement. The questionnaire is meant to be completed in the GP’s waiting room by patients after consultation with their GP.

The 18 questions which concentrate on the patient’s background concern the patient’s socio-economic status, perceived health, reason for visiting the GP, and visits to medical specialists and hospitals. Six questions deal with measuring experiences with ‘continuity of care’, e.g. the use of medical records. ‘Quality of care’ as experienced by patients is measured in 13 questions (e.g. about the satisfaction of care needs in connection to the patient’s relationship with the GP, aspects of communication, safety, complaint handling and preventive activities).

As in the GP questionnaire, the 14 questions about the ‘accessibility of care’ can be divided into physical and financial access. These questions also include the time the GP has available for the patient, the availability of home visits and waiting times. Three questions pay attention to ‘equity in access’ and one question to ‘equity in treatment’. ‘Coordination’ is measured with five questions on experiences of coordination in the case of referral and on treatment by a practice nurse.

To mirror the questions in the GP questionnaire about autonomy, patients are asked about their involvement in decision making and referrals. ‘Comprehensiveness of services’ is mirrored in a question about patients’ views on the breadth of the clinical task profile of services offered by the GP. Finally, two questions specifically related to avoidable hospitalisation were included.

Major sources for this questionnaire were the Consumer Quality Index for GPs,14 the EUROPEP,[15] several international Commonwealth Fund questionnaires [1620] and Starfield’s Adult Primary Care Assessment Tool.[21] Compared with the GP questionnaires, more questions for patients were identified in the domain of equity in access and treatment. As few questions were found on patient autonomy, new questions had to be developed on this theme.

Patient Values questionnaire

Next, a Patient Values questionnaire was developed. Measuring what patients consider important enables the weighting of their experiences.[22] The Patient Values questionnaire contains [19] questions (seven of which have three or more subquestions). Again, most questions are statements with multiple choice answers. A few questions ask the patient to choose from a list what they consider most important and fill in a number.

The 12 questions asking about the patient’s background are similar to those in the Patient Experiences questionnaire. Three questions contain statements asking patients about the importance of certain aspects of care (e.g. ‘How important is it that the practice has extensive opening hours?’).

Finally, four questions focus on communication between GPs and patients. The statements in these questions were developed by the GULiVer partnership based on their research on ‘tips’ from lay people on how medical consultations could become more successful from their perspective.[23]

Practice questionnaire

A 12-question practice questionnaire was developed to record the response rate among patients during the implementation of the survey and to measure practice- related indicators with regard to the communication of opening hours, and equity in access (e.g. for handicapped persons). Most questions were based on the European Practice Assessment indicators.[24]


The four questionnaires have been developed to characterise the organisation and delivery of primary health care and to compare and analyse its outcomes. The development of questionnaires for a multicountry study on broad themes such as quality, costs and equity in primary care requires a balance between methodological requirements and practical feasibility. Indeed, all dimensions deserved to be thoroughly investigated, although theymay be difficult tomeasure reliably, but it must be accepted that only a limited set of questions can be asked.

Nonetheless, the QUALICOPC consortium has been able to produce the four questionnaires—as far as possible—based on existing, validated questionnaires and tested through a pilot survey in three countries. A limitation of the pilot survey is that it was carried out in only three countries. However, much attention has been paid to having valid translations in each language. In each country, an official back-and-forth translation procedure is used for the questionnaires, in which translators are asked to take comprehensibility into account. Another limitation of the questionnaire development is that questions, derived from various validated sources, often had to be ‘processed’ to make them suitable for the QUALICOPC study. This may have resulted in a loss of validity and needs to be taken into account in the analysis phase.

The questionnaires for GPs and patients contain questions that go beyond general practice. Furthermore, data about primary health care (e.g. about its costs) will be gathered at the national level in available databases. Nevertheless, results regarding quality of primary health care as a whole need to be interpreted with care.

The dimension ‘Governance’ has not been covered in any of the questionnaires, because aspects of governance are relatively distant from daily reality in primary health care. However, information on governance will be used and derived from the PHAMEU database. Relatively new topics that will be explored in the QUALICOPC study are equity in access and treatment, patient autonomy, disease management, avoidable hospitalisation and patient experiences with primary health care in general. There are also aspects of care which might be interesting, but are not included in these questionnaires. This included new developments around telemedicine, but also the experiences of patients around disease management programmes. Equity in health can also not be measured through this survey, as we only include patients who visit GP practices and, moreover, we do not measure health outcomes. Several questions had to be omitted to keep the length of the questionnaire reasonable.

Because the sources were identified from Western countries, the questionnaires that we developed are more likely to be suitable for use in Western countries than in others. However, the 35 countries in which the questionnaires will be used in the context of the QUALICOPC study match this profile well. The results of the study will add to the available evidence on the relationship between the strength of primary care systems and their outcomes. The data from the 35 countries will be linked to the practices and their patients. Analyses of the data will provide insight into variations between countries at the level of the patient, GP practice and country. The patient questionnaires may also be suitable for use at the practice level by GPs to analyse developments in the GP practice by inviting a sample of patients every year to complete a questionnaire.


0.329+ Million Readerbase
Share This Page
Recommended Webinars & Conferences