Quality in Primary Care Open Access

Keywords

appraisal, community pharmacist, con-sultation, general practitioner, patient centred, Peer Review

Introduction

It has been clearly established that the delivery of effective doctor–patient communication not only en-hances patient satisfaction, but can also lead to improved clinical outcomes.[1] By contrast, poor communication can adversely affect clinical outcomes, patient satis-faction and levels of litigation.[2] Communication skills and the ability to consult effectively are key to the delivery of quality patient care. This has been recog-nised by the General Medical Council (GMC)[3] and the Royal College of General Practitioners (RCGP),[4,5] and is one the domains requiring evidence in the General Practitioner Appraisal and Revalidation Framework.[6] The introduction of appraisal in the UK and the debate over what evidence will be necessary to satisfy the requirements of revalidation[7] have raised the ques-tion of how doctors can access education in this area and provide objective and verifiable evidence of their ability to communicate effectively with patients.[8]

Owing to their new contractual obligations, com-munity pharmacists in the UK[9] are working in a rapidly evolving and changing clinical environment and are now expected to provide increasingly complex advice to patients in the primary care setting. To date, review of and training for community pharmacists in communicating with patients about clinical issues has been minimal.[10]

Since Balint[11] in the 1950s, many doctors and physiologists[12–15] have analysed the doctor–patient con-sultation. Perhaps the work carried out by Pendelton et al[16] in the 1980s was most influential because he was the first to describe a method of teaching the consul-tation using video. This has influenced GPs in training for the last two decades, and consequentially a large proportion of GPs working in the UK today have learned about patient-centred consulting using this method. It has, however, been established that once communication skills have been developed they need to be practised or they may wither.[17]

One method of obtaining evidence about the qual-ity of communication skills in the primary care setting is through external Peer Review. Peer Review has been described as the critical evaluation of a specific aspect of a practitioner’s performance by professional col-

leagues, facilitated by a structured instrument whose utility reflects the purpose for which it is intended.[18,19]

External Peer Review has been described as an essential stimulus to effective performance and it has been suggested that it should become an integral part of self-regulation.[20]

In the west of Scotland, established GPs have been able to voluntarily submit videotaped consultations for external Peer Review as part of their continuing personal development (CPD) for 10 years (Box 1). A consultation Peer Review group (CPRG) has been established. It has become clear to the reviewers that consultations submitted for review by established GPs differ from those submitted for assessment by GPs in training. The consultations tend to be more realistic, often more complex, everyday general practice which did not fit into the performance criteria-based assess-ment tool used by Membership of Royal College of General Practitioners (MRCGP) examination. The 18 active reviewers in a CPRG are trained in assessing the quality of consultations and writing individual, de-scriptive feedback using a tool based on a patient-centred model.[21] However, this instrument has not previously been subject to adequate evaluation. This study describes the first steps in the development and preliminary psychometric evaluation of a generic tool to be used to inform feedback about both GP consul-tations and clinical discussions between pharmacists and patients in the primary care setting.

Method

Development of tool

Four GP educationalists, with experience in assessing GP consultations, used their professional and teaching expertise, together with accumulated knowledge of patient-centred consulting,[14–16] to analyse the content of patient–professional interactions. By round the table discussion and professional consensus, relevant questions were agreed and collated. The aim was to be as inclusive as possible, as it was acknowledged that poor questions could be detected and removed, but there was no recompense for any items not initially included. These questions resulted in development of items for proposed feedback instrument.

Content validity exercise

To demonstrate that the important areas of the con-sultation were being considered in the proposed instru-ment a content validity exercise[22] (CVE) was performed. The proposed instrument was sent, by post, to a convenience sample of 10 GPs and pharmacists. All recipients were well informed about consulting techniques and work in primary care in Scotland. They consisted of six GPs from the existing group of Peer Reviewers, two GPs with a special interest in consul-tation skills, together with two primary care pharma-cists with a background in CPD. All 10 professionals approached to take part in CVE agreed to provide responses. Each respondent rated the perceived rele-vance of each item to a patient-centred consultation using a Content Validity Index (see Appendix A online at  www.radcliffe-oxford.com/journals/J10_Quality_  in_Primary_Care/Supplementary_Papers.htm>) and returned their response by post. This rating exercise underwent an iterative process until 80% agreement for each item of the proposed instrument was demon-strated. The identified items formed the content of the feedback instrument to be tested further. A seven-point Likert-type[23] rating scale was agreed for each item to enhance reliability in discrimination.

Item correlation pilot

To assess the contribution of each item in the proposed instrument and to establish consistency, an item corre-lation pilot (ICP) was performed. Multicentre Ethical Approval was obtained. A convenience sample of six GPs, who had voluntarily submitted consultations for Peer Review under the existing CPD mechanism, consented to take part in this project.

The authors ran workshops at several pharmacy educational meetings, with the aim to identify six volunteer pharmacists who were prepared to make and submit videotapes of their consultations with patients. GPs and pharmacists obtained informed consent from all participating patients both before and immediately after consultation.

All established consultation Peer Reviewers in the CPRG, who were not involved with the development of the instrument, were invited to take part in reviewing consultations for this project. Six volun-teered and were selected for the task.

At a central venue, three reviewers assessed and scored the same three consultations on each of the six GP tapes, and three other reviewers assessed and scored three identical consultations on each of the pharmacist tapes.

The following statistical analysis was undertaken:

flassessors – Limits of agreement were calculated to give a range within 95% of the differences between which the assessors’ scores would lie, both for those assessing GP consultations and those assessing phar-macist consultations.

• GP consultation skills – The score each GP was given for each item by each of the three assessors was plotted for each consultation.

Results

It was acknowledged at the time of initial development of the instrument that pharmacists do not examine patients. Although an identical generic tool could not be employed to assess the patient centredness of both GP and pharmacist consultations, by simply removing the two items relating to patient examination in the pharmacy tool, two similar assessment tools were able to be developed and tested concurrently (see Appendices B(GP) and B(Ph) online at  www.radcliffe- oxford.com/journals/J10_Quality_in_Primary_Care/  Supplementary_Papers.htm>).

Content validity exercise

• The proposed GP instrument consisted of 24 items each rated on a seven-point Likert scale, 23 ques-tions relating to the patient-centred consultation and 1 giving an overall score.

• The proposed pharmacist instrument consisted of 22 of these 24 items each rated on a seven-point Likert scale, 21 questions relating to a patient-centred consultation and 1 overall score.

flan agreement of 80% was demonstrated for in-clusion of all 24 items using the proposed tool after one round of the CVE. There were some minor adjustments made to the wording of individual items as a result of this exercise.

Item correlation pilot

GP consultations

Three consultations from each of six GP tapes were viewed independently by three reviewers, i.e. each re-viewer watched and assessed 18 identical consultations.

STATISTICAL ANALYSIS OF GP TAPES

•  Assessors:

– Inter-rater consistency was measured by assessing the agreement between rater’s scores for each item included in the instrument. The scores of assessor 1 and assessor 2 were within 1 point (on the Likert scale) of each other for 83% of the items. Assessors 1 and 3 agreed within 1 point for 76% of the items and assessors 2 and 3 agreed within 1 point for 63% of the items. No assessor differed on average from the other two

by more than 0.7; there was good agreement between assessors.

• GP consultation skills (Table 1):

– Reviewer’s qualitative judgements regarding those GPs who performed exceptionally well (GP5) and those who performed less well (GP6) were backed up by a statistical difference in their mean scores:

(i) GP5 mean = 5.36, GP6 mean = 3.45

(ii)  Difference in mean = 1.91

(iii) Confidence interval (CI) for difference = 1.73 – 2.09, P < 0.0001.

Pharmacist consultations

Only four pharmacists were able to submit tapes in the allotted period, three consultations from each of four pharmacist tapes were viewed independently by three GP reviewers, i.e. each GP reviewer watched and analysed 12 identical consultations.

STATISTICAL ANALYSIS OF PHARMACIST TAPES

• Assessors:

assessor 1 and assessor 2 were within 1 point (on Likert scale) of each other for 77% of the items. Assessors 1 and 3 agreed within 1 point for 67% of the items and assessors 2 and 3 agreed within 1 point for 73% of the items.

• Pharmacist consultation skills (Table 2):

– A simple mean of the scores for each pharmacist was used to summarise their overall perform-ance. The individual performances were fairly similar, with no pharmacist appearing to per-form exceptionally well or poorly. The numerical values are lower than those noted in equivalent GP assessment.

Discussion

Summary of main findings

This study has outlined the groundwork and first steps in developing and testing an instrument to be used to deliver effective, consistent written feedback for established GPs and community pharmacists working in the general practice setting. Previous studies have focused on assessing GP consultation competence– Inter-rater consistency was measured by assess- tablished GPs and community pharmacists working ing the agreement between rater’s scores for each in the general practice setting. Previous studies have item included in the instrument. The scores of focused on assessing GP consultation competence using standardised patients,[24,25] or within the training environment.[26] Although the initial aim was to de-velop a generic tool, two similar assessment tools were developed and tested, one to rate the patient-centred consulting skills of GPs and the other to rate the patient-centred consulting skills of pharmacists.

The aim of consultation Peer Review is to provide effective written feedback, within the limitations of this preliminary, small study the content of both tools was demonstrated to show certain aspects of both face validity and content validity. To underpin consistency in this feedback, it is necessary to establish reliability of the instrument. This study has begun to explore aspects of internal consistency and inter-rater reliability. The results, although limited, indicate that when the tool was applied to the GP tapes, the reviewers demon-strated concordance. It was also indicated that a simple mean of the recorded scores could consistently rep-resent the GPs’ overall performance in patient-centred consulting and that this was consistent over three consultations. When the simple mean was compared with the informed judgement of the reviewers it was possible to discriminate between the differing levels of skill demonstrated by the GPs. Agreement was less evident in the pharmacist tapes, evaluation of per-formance was less consistent and generally the scores were lower and more widespread.

The initial method proposed the inclusion of a reliability pilot, to be undertaken once the ICP was completed and analysed, to assess the ability of trained reviewers in using the instrument in a consistent manner. However, analysis of the ICP demonstrated that the level of agreement differed when comparing GP tapes and pharmacist tapes; being significantly greater when used for the GP tapes. This evidence demonstrated that the tool did not discriminate at consistent levels with each professional group, and therefore the reliability pilot was not pursued.

Sensitivity analysis suggested that several of the items used in each tool could be omitted with no effect on overall sensitivity of the instrument. How-ever, to enhance the writing of consistent, effective, constructive feedback the authors feel it is important for the reviewers to consider all areas of the consul-tation.

Strengths and limitations of study

NHS Education for Scotland (NES) aims to encourage health professionals to develop interprofessional learn-ing.[27] In this study, GPs and pharmacists worked together achieving strong links and understanding between the professions.

The study took place in Scotland, pharmacists and GPs were recruited from the west of Scotland, re-viewers and those involved in CVE were from all areas of Scotland. The consultation is a core element of primary care in the UK and these results are therefore relevant to all areas of the UK.

The instruments were initially developed and tested only by GPs. The absence of pharmacist involvement in the initial question development may have biased the results of the CVE. The authors were involved in the training of postgraduate pharmacists in com-munication skills at the time of the study and were aware that relevant expertise was not available within the pharmacy profession in Scotland. The ‘experts’ employed were highly experienced in consultation skills training, but the number of individuals with sufficient knowledge and expertise to be deemed appropriately skilled in this area is limited.

Both GPs and pharmacists submitting their consul-tations for this project were volunteers. All assessors were GPs, at the time of the study only GPs had been trained in consultation Peer Review; this meant that the phar-macist consultations were assessed from a GP per-spective. Assessors were all members of the established CPRG, and as such were both confident and experi-enced in reviewing consultations. Non-trained indi-viduals would undoubtedly have been less consistent.

Available resources allowed only small numbers of participants to be included in this pilot. Only four pharmacists were able to submit consultations for review within the allotted time. Larger numbers of consultations or larger numbers of assessors may have influenced the results.

Each assessor was asked to consider 22 or 24 individual questions about each consultation, a task, which requires expertise and prolonged concentration, sufficient resources were identified to ensure pro-tected time and all assessments were undertaken in a venue with the required facilities.

The pharmacists

Until now, few pharmacists have received structured training or assessment in consultation skills,[28] and this topic has only recently been tentatively introduced into their undergraduate curriculum. By contrast, UK medical students have been taught communication skills as part of their core curriculum since the mid-1990s.[29] GP registrars have been taught consultation skills since the early 1990s, and these skills have been assessed summatively since 1996.[30]

The results suggested that the tools employed demonstrated certain aspects of reliability when ap-plied to participants who had received teaching in patient-centred consulting skills. Reliability was more limited for those participants who had not received teaching in this area. These outcomes have led to the introduction of training in patient-centred consulting and the development of a consultation Peer Review process for pharmacists throughout Scotland led by GPs and funded by NES.

The GP context

Communication with patients is one of the five core categories, to be undertaken on a five-year cycle, in GP appraisal in Scotland.[31] However, it has been demonstrated that fewer than 10% of those appraised and submitted evidence in this area felt there had been any impact on their consulting as a result.[32] This requirement can be addressed in a variety of ways including the use of standardised survey, e.g. General Practice Assessment Questionnaire (GPAQ). Surveys such as GPAQ are a required annual element of the Quality and Outcomes Framework. Although their value has been questioned, it is not surprising that most GPs chose this method, this instrument, how-ever, has not been formally tested for reliability.[33] The authors suggest that participation in an effective Peer Review process may have greater educational and developmental impact. Peer Review of consultations may also offer a more feasible method of demonstrat-ing consultation skills for sessional doctors or those in small practices, than the proposed reliance on multi-source feedback.[34]

The GMC Good Medical Practice Framework for Appraisal and Assessment (Revalidation: the way ahead) for all doctors proposed that ‘validated tools for feedback about doctors’ practice’ could provide the evidence required for appraisal in all four domains (Box 2).[31] This is an example of initial development of such a tool.

of performance. It is essential to also consider accept-ability, feasibility and educational impact[35] both for participants and reviewers. Further work is under way to evaluate this.

If this model was to be adopted by community pharmacists, more work, including retesting would be required, using consultations from pharmacists who have gained experience in patient-centred consulting.

When providing feedback to peers, the largest element of variability, in common with other instru-ments, derives from the difference between individual assessors.[36,37] Significant additional work is required to ensure acceptable reliability of the instrument, but in common with other assessment instruments[38] ongoing calibration of assessors is essential to optimise inter-rater reliability.

Previous work has suggested that because the con-sultation is the cornerstone of medical practice, a central focus of revalidation should be the assessment of consultation competence,[39] and discussion of the new regulatory processes suggests that all doctors may be asked to provide evidence of the effectiveness of their consultation skills for the purposes of revalid-ation.[7] The results suggest that with further develop-ment this tool could offer a method of objectively and reliably assessing a GP’s performance in the consulting room. It would, however, be imperative to ensure that adequate resources were available to provide training in this area, to enable consistent feedback and that effective support was available for those doctors whose performance gave cause for concern.

The future

The key purpose of consultation Peer Review is to facilitate educational feedback to GPs and pharmacists on their demonstrated consulting skills. The results of this pilot study highlight the validity and potential reliability of this tool in providing consistent Peer Review feedback for GP consultations.

Considerable further work is required to confirm the overall utility of the instrument. A simple mean of scores for GPs summarised their overall performance, and was able to discriminate between different levels

The authors would like to thank Dr Douglas Murphy and Dr Colin Hodgson for their help in developing the content of the instrument, the CPRG for their reviews and all participating GPs, pharmacists and their patients for the consultations provided for review. Grateful thanks to Dr Lillian Murray, University of Glasgow for statistical advice.

Acknowledgements

The authors would like to thank Dr Douglas Murphy and Dr Colin Hodgson for their help in developing the content of the instrument, the CPRG for their reviews and all participating GPs, pharmacists and their patients for the consultations provided for review. Grateful thanks to Dr Lillian Murray, University of Glasgowfor statistical advice.

References

1. Simpson M, Buckman R, Stewart M et al. Doctor– patient communication: the Toronto consensus state-ment. > British Medical Journal 1991;303:1385–7.
2. Health Service Ombudsman for England. Annual Report 2004–5. London: HMSO, 2005.
3. General Medical Council. 2001 Good Practice. London: General Medical Council, 2001.
4. Royal College of General Practitioners. The General Practice Curriculum. The General Practice Consultation. London: RCGP, 2007.
5. Royal College of General Practitioners. Good Medical Practice for GPs. London: RCGP, 2008.
6. General Medical Council. Good Medical Practice Frame-work for appraisal and revalidation, 2011. www. gmc-uk. org/doctors/revalidation/revalidation _gmp_framework. asp (accessed 29 July 2011).
7. Donaldson L. Medical Revalidation – Principles and Next Steps: the report of the Chief Medical Officer. London: Department of Health, 2008.
8. General Medical Council. Working Framework for Enhanced Appraisal and Assessment, 2008. > www.gmc- uk.org/about/reform/gmp_framework.asp
9. Department of Health. Implementing the New Com-munity Pharmacy Contractual Framework. London: De-partment of Health, April 2005.
10. Johnson S, Hughes E and White R. Developing com-munication skills for pharmacist led clinics. > Medical > Education 2002;36:1092–3.
11. Balint M. The Doctor, His Patient and the Illness. London: Pitman Medical,1957.
12. Byrne PS and Long BEL. Doctors Talking to Patients. London: RCGP Publications, 1976.
13. Stott NC and Davies RH. The exceptional potential in each primary care consultation. Journal of Royal College of General Practitioners 1979;29(201):201–5.
14. Neighbour R. The Inner Consultation: how to develop an effective and intuitive consulting style (2e). Oxford: Radcliffe Publishing, 2005.
15. Silverman JD, Kurtz SM and Draper J. Skills for Com-municating with Patients. Oxford: Radcliffe Publishing, 1998.
16. Pendleton D, Schofield T, Tate P and Havelock P. The Consultation: an approach to learning and teaching. Oxford: Oxford University Press, 1984.
17. Skelton J. Everything you were afraid to ask about communication skills. BMJ 2005;55:40–5.
18. Grol R. Quality improvement by Peer Review in primary care: a practical guide. Quality and Safety in Health Care 1994;3(3):147–52.
19. Evans R, Elwyn G and Edwards A. A review of instru-ments for peer assessment of physicians. BMJ 2004; 328:1240–3.
20. Irvine D. The performance of doctors. 1: Professional-ism and self regulation in a changing world. British Medical Journal 1997;314:1540–5.
21. Stewart M, Brown JB, Weston WW, McWhinney IR, McWilliam CL and Freeman TR. Patient-centred Medi-cine, Transforming the Clinical Method (2e). Oxford: Radcliffe Publishing, 2003.
22. Lynn MR. Determination and quantification of content validity. Nursing Research 1986;35(6):382–5.
23. Likert R. A technique for the measurement of attitudes. Archives of Psychology 1932;22(140).
24. Fraser RC, McKinley RK and Mulholland H. Consul-tation competence in general practice: testing the re-liability of the Leicester assessment package. > British > Journal of General Practice 1994;44:293–6.
25. Fraser RC, McKinley RK and Mulholland H. Consul-tation competence in general practice: establishing the face validity of prioritized criteria in the Leicester as-sessment package. > British Journal of General Practice 1994;44:109–13.
26. Campbell LM, Howie JGR and Murray TSM. Summative assessment: a pilot project in the west of Scotland. > British Journal of General Practice 1993; 43:430–4.
27. NHS Education for Scotland, 2005. > www.nes.scot. nhs.uk/disciplines
28. Waterfield J, Aspinall V and Hall S. The use of digital video media in teaching of communication skills. The Pharmaceutical Journal 2009;282:135–6.
29. General Medical Council. Tomorrow’s Doctors. London: GMC, 1993.
30. Campbell LM and Murray TS. Summative assessment of vocational trainees: results of a 3 year study. > British > Journal of General Practice 1996;46:411–14.
31. NHS Education for Scotland. www.scottishappraisal. scot.nhs.uk/appraisal-preparation/core-category (ac-cessed 18 November 2010).
32. Colthart I, Cameron N, McKinstry B and Blaney D. What do doctors really think about the relevance and impact of GP appraisal 3 years on? > British Journal of > General Practice 2008;58(547):82–7.
33. Harkins M, Fraser A, Hodson A, Hooley C and Smith H. Measuring patient satisfaction for the Quality and Out-comes Framework. > British Journal of General Practice 2007;57(542):737–40.
34. Pringle M. Regulation and revalidation of doctors. British Medical Journal 2006;333:161–2.
35. General Medical Council. Revalidation: the way ahead, 2010. > www.gmc-uk.org/the way ahead
36. Ram P, Grol R, Rethans JJ, Schouten B, van der Vleuten C and Kester A. Assessment of general practitioners by video observation of communicative and medical per-formance in daily practice: issues of validity, reliability and feasibility. > Medical Education 1999;33:447–54.
37. Roberts C, Cromarty I, Crossley J et al. The reliability and validity of a matrix to assess completed reflective personal development plans of general practitioners. > Medical Education 2006;40:363–70.
38. Ramsay PG and Wenrich MD. Peer ratings: an assess-ment tool whose time has come. > Journal of General > Internal Medicine 1999;14:581–2.
39. McKinley RK, Fraser RC and Baker R. Model for directly assessing and improving clinical competence and per-formance in revalidation of clinicians. > BMJ 2001; 322:712–15.

Contributors

RM led the study conception and design, and orchestrated the research application, subject partici-pation, the consultation review process and the data collection. AP coordinated recruitment of pharma-cists. NC contributed considerably to both writing of the paper and to revision of the manuscript.

Funding

This study was funded by NHS Education for Scotland.

Ethical Approval

This study was approved by Scotland A Research Ethics Committee, Edinburgh.

Peer Review

Not commissioned; externally Peer Reviewed.

Conflicts of Interest

NC is GP National Appraisal Adviser for Scotland. RM manages Consultation Peer Review Group.