Quality Improvement Report - (2008) Volume 16, Issue 1
Assistant Professor, Family and Community Medicine, King Saud Ibn AbdulAziz University for Health Sciences, Riyadh, Saudi Arabia
Received date: 19 June 2007; Accepted date: 10 July 2007
Background Diabetes constitutes a major burden of disease globally. Both primary and secondary prevention need to improve in order to face this challenge. Improving management of diabetes in primary care is therefore of fundamental importance. Objective The objective of these series of audits was to find means of improving diabetes management in chronic disease mini-clinics in primary health care. In the process, we were able to study the effect and practical usefulness of different audit designs – those measuring clinical outcomes, process of care, or both. Setting King Saud City Family and Community Medicine Centre, Saudi National Guard Health Affairs in Riyadh city, Saudi Arabia. Methods Simple random samples of 30 files were selected every two weeks from a sampling frame of file numbers for all diabetes clients seen over the period. Information was transferred to a form, entered on the computer and an automated response was generated regarding the appropriateness of management, a criterion mutually agreed upon by care providers. The results were plotted on statistical process control charts, p charts, displayed for all employees. Data extraction, archiving, entry, analysis, plotting and design and preparation of p charts were managed by nursing staff specially trained for the purpose by physicians with relevant previous experience. Results Audit series with mixed outcome and process measures failed to detect any changes in the proportion of non-conforming cases over a period of one year. The process measures series, on the other hand, showed improvement in care corresponding to a reduction in the proportion nonconforming by 10% within a period of 3 months. Non-conformities dropped from a mean of 5.0 to 1.4 over the year (P 0.001). Conclusion It is possible to improve providers’ behaviour regarding implementation of given guidelines through periodic process audits and feedbacks. Frequent process audits in the context of statistical process control should be supplemented with concurrent outcome audits, once or twice a year.
audit, diabetes, outcome, primary care, process
Diabetes mellitus contributes much to the global burden of disease. Proper management and followup of diabetes at primary care level can play a pivotal role, both in primary and secondary prevention.
Glycaemic control in type 2 diabetes in primary care is generally reported to be poor,[1] and treatment approaches have been found not to be intensive enough for a large proportion of patients, especially those with longer duration of disease.[2] Treatment regimes at this level of care are not appropriate to achieve the required targets of care.[3]
Lack of appropriate adjustments in treatment in response to poor control indicators, termed clinical inertia, has been reported to be a problem at different levels of the hierarchy of care for diabetes.[4,5] The primary care environment is especially prone to lead to burnout,6 resulting in inappropriate therapeutic choices and affecting outcomes in the long run.
Care providers and organisations need to find means to overcome clinical inertia, ensuring adjustment of therapy when required and the use of insulin when clinically indicated. Educating general practitioners (GPs) in this regard improves the control of diabetes without significantly increasing healthcare costs.[7]
In Saudi Arabia, substantial variation in the quality of primary care services has been reported, with a need to improve the management and organisation of primary care services and implementation of professional development strategies to improve the knowledge and skills of staff.[8]
Clinical audits, monitoring the care delivery process, can provide learning opportunities for care providers through interaction with a self-managed objective evaluation process. Ongoing performance feedback based on audit results may prove to be an important tool for both mutual learning and clinical performance modification and improvement. Such feedback has been shown to improve provider behaviour, leading to better clinical outcomes and improved diabetes control with reduced complications.[9–11] The current study presents the results of two audit series carried out in the diabetes care clinic at King Saud City Family and Community Medicine Centre, one of the primary care portals of Saudi National Guard Health Affairs, Riyadh, Saudi Arabia. The first series of audits was conducted from March 2003 to February 2004, and the subsequent one from October 2004 to September 2005. Both series consisted of fortnightly audits, with the first one being mixed process and outcome audits, while the second one consisted of pure process audits.
We want to share our experience that outcomes are slow to develop for any efforts aimed at improvement; it is more rewarding in the short term, and perhaps more effective in the long run, to focus on pure process audits. These may be supported by once- or twice-ayear outcome audits.
The Saudi Arabian National Guard Health Affairs provides primary, secondary and tertiary care to National Guard employees and their families. The primary care component is provided by the Department of Family and Community Medicine. In Riyadh city, three main centres provide such care, King Saud City centre being one of them.
King Saud City housing consists of about 1250 residential units. Primary care and family medicine services in this housing are provided by 23 GPs distributed over three satellite centres. Each one of these centres, in addition to primary care and family medicine facilities, runs chronic disease mini-clinics caring for individuals with diabetes, hypertension, dyslipidaemia and asthma. Each clinic is run by a family physician (GP), with the provision of referral to secondary care if necessary. More than 100 patients are seen in each mini-clinic every month.
Concerns about quality of care in these clinics have been voiced since the early 1990s, and occasional clinical audits were also carried out, a routine whose futility we recognised in the late 1990s.
We started regular, periodic audits in 2003. The results of these audits were studied after putting them in the context of statistical process control, taking into account random variation, against the given standards of performance set in consultation with the management.
Both series consisted of fortnightly audits. This interval was decided based on the work volume of the clinics, which was sufficient to provide samples of 30 files every two weeks. The sample size for these auditswas decided based on the population parameter of clients who were not provided appropriate care (non-conforming) set at 80% and 70%. These parametric targets were set in consultation with the management based on pervious clinical impressions and on what was perceived to be acceptable for all involved in the process of care. This number satisfies the condition for sufficiency of sample size as all n(p) and n(1 – p), i.e. 30(0.7), 30(1–0.7), 30(0.8), 30(1 – 0.8), are above 5.[12] From a sampling frame of all diabetes care clients seen over the previous two weeks, a simple random sample of 30 files was selected. In the beginning, randomisation was done through using a random numbers table and dice, a process that was found to be rather cumbersome. This was later replaced with a computer program developed and tested locally by physicians. Nurses working in the chronic diseaseminiclinics were trained for this purpose. From the selected files, data were extracted by the trained nursing staff, and a data form was filled in for each file. A computer database was created using Epidata,[13] which, upon data entry, automatically classified each record as conforming or non-conforming to the given criterion of appropriate care. The proportion of patients nonconforming to the given criterion was calculated and plotted on an attribute control chart (or p chart). All stages of the process were managed by trained nursing staff.
The criterion of appropriate care for both the series was decided collaboratively by physicians involved in the process of care. Attribute control charts, p charts, were designed by physicians and prepared by the nurses trained for the purpose. The charts were designedwith the centre line, specifying the parameter at 0.8 and 0.7 for the two audits respectively, and upper and lower control lines drawn at + and –3
standard errors, based on the formula:
For a parameter of 0.8, with a sample size of 30, the upper control limit was calculated to be 1.0 and the lower one 0.5809, while for 0.7, the limits will be 0.9510 and 0.4490.
Both series of audits were carried out in the same practice with the same physicians providing care throughout the period.In the first series, consisting of 23 audits, both haemoglobin A1c (HbA1c) test frequency and results were recorded for each case. The criterion of appropriate care depended on both the variables, the test having been done at least once every six months and HbA1c level being 7% or below. In case HbA1c was recorded within the previous six months, each case was classified as adequately (HbA1c < 7.0%), suboptimally (HbA1c 7.0–8.0%), or inadequately (HbA1c > 8.0%) controlled.
In the second series, consisting of 19 audits, eight components of clinical diabetes care were measured, the criterion of appropriate care being simultaneous provision of all the eight components: presence of follow-up form, completion of follow-up form,HbA1c measurement within the previous three months, and microalbuminuria, fundoscopy, lipid profile, diabetes education and flu shot advice every year (see Tables 1 and 2).
Table 1: List of variables, diabetes clinics audits: Series 1 King Saud City Family and Community Medicine Centre, National Guard Health Affairs, Riyadh, March 2003 to September 2004
Table 2: Diabetes clinics audits: Series 2 October 2004 to September 2005
There was no targeted, systematic feedback of audit results to the care providers although the process control charts were displayed where they were visible to all employees including the physicians involved in care.
Statistical analysis was done with Stata Version 8.2,14 and EpiInfo version 6.04.15 Ninety-five per cent confidence intervals (CI) were used for parametric estimation. Two-tailed t tests were used for two-group comparisons of numerical variables.
Most relevant information can be extracted from a p chart just by an ‘eyeball test’; the interpretation being that as long as the sample statistics values are randomly distributed around the centre line within the control limits, the process is under control for the said parametric specification.
In the first series chart (see Figure 1), the process was under control around the centre line at 0.8. In other words, the null hypothesis that the population proportion of non-conforming cases is 80% could not be rejected.
Figure 1: Attribute chart (p chart) for proportion of cases not conforming with the given criteria in the first series of audits March 2003 to February 2004. LCL, lower control limit; UCL, upper control limit
The second series started with the process out of control for the given parametric specification of 0.7, but came down to stay within control limits around the said specification after five audits.
The first series consisted of 634 cases studied over the period of one year. Fifty-five percent of those studied during the first series were males, while 45% were females. Difference in non-conformities between the two sexes during this series was not significant (P = 0.1). Difference in proportion non-conforming between the sexes was also not significant (P = 0.5).
This series failed to detect any improvement in the proportion of patients non-conforming with the criterion of good care over a period of one year (t test for difference between meannumber of non-conformities per case for the first and the last audit: P = 0.05; Cuzick’s trend test Z = 2.64, P = 0.008; see Table 3 and Figure 1).
Table 3: Average number of examination non-conformities per case by audits in the first series of audits March 2003 to February 2004
The number of non-conformities for the first series of audits ranged from 0 to 2 per case, with a mean of 1.1 0.6. However, the mean number of non-conformities, 1.23 in March 2003, reduced to 0.97 in February 2004 with some borderline significance (P = 0.05), but trend analysis showed a trend for the mean non-conformities per case to increase.
The second series of audits consisted of 557 cases studied over a period of one year. Among those studied 58.5 % were females while 41.5 % were males.The second series of audits consisted of 557 cases studied over a period of one year. Among those studied 58.5 % were females while 41.5 % were males. Figure 1 Attribute chart (p chart) for proportion of cases not conforming with the given criteria in the first series of audits March 2003 to February 2004. LCL, lower control limit; UCL, upper control limit Audit number Proportion non-conforming Table 2 Diabetes clinics audits: Series 2 October 2004 to September 2005 Variable Variable type Type of measure 1 File number Binary (yes/no) Identification 2 Age Binary (yes/no) Identification 3 Sex Binary (0 male 1 female) Identification 4 Presence of follow-up form Binary (yes/no) Process 5 Completion of form Binary (yes/no) Process 6 HbA1c within 3 months Binary (yes/no) Process 7 Microalbuminuria within 1 year Binary (yes/no) Process 8 Fundoscopy within 1 year Binary (yes/no) Process 9 Lipid profile within 1 year Binary (yes/no) Process 10 Education within 1 year Binary (yes/no) Process 11 Flu shot advice within 1 year Binary (yes/no) Process Statistical process control for diabetes care in primary care 57 The number of non-conformities or proportion nonconforming did not differ significantly between the two sexes (P = 0.2 and 0.1, respectively). This series detected improvement in the process of care after the initial three months (t test between mean number of non-conformities per case for the first and the last audit: P < 0.001; Cuzick’s trend test for mean nonconformities per case: Z = –13.5, P < 0.001, see Table 4, and Figure 2).
Table 4: Average number of examination non-conformities per case by audits in the second series of audits October 2004 to September 2005
Figure 2: Attribute chart (p chart) for proportion of cases not conforming with the given criteria in the second series of audits October 2004 to September 2005. LCL, lower control limit; UCL, upper control limit
The number of non-conformities for the second series of audits ranged from 0 to 8 per case, with a mean of 2.4 1.9. The mean number of non-conformities, 5.1 in October 2004, reduced to 1.4 in September 2005 (P < 0.001).
Using a mixed measures audit, a whole year’s process of measurement failed to detect any improvement in care. On the contrary, there was a trend towards an increase in the mean number of non-conformities per case. Because of the audit being a mixed one, it was difficult to judge, based on the data-management structure, whether this lack of improvement was primarily because of a lack of process or an outcome response.
Although a process may be more sensitive to change as compared to an outcome, a sluggish outcome response will mask any improvement in process too.
For purely process measure audits, on the other hand, it was possible to demonstrate a reduction of 10% in the proportion of non-conforming cases within three months of starting the audits, a statistically significant change. This reduction in the number of non-conformities and the overall proportion nonconforming with the given criteria was not the result of any specially focused feedback or individual meetings with the care providers, the only intervention being the care providers’ awareness of the process of audit and graphic presentation of results.We presume that a more individual or group-specific customised feedback and dialogue may have produced even better results.
This was not a study designed to compare the results of the two audits as such, and we do not claim to have used a systematic, validated comparison process. The objective of the audit initiative in the context of statistical process control was to improve the management of diabetes in the practice. Having failed to make a difference the first time, we adjusted the audit design and noted the change not only in the care providers’ behaviour but also in the interest and commitment of those involved in the audit process. All this was achieved without any special feedback. We believe the difference we noticed was significant enough to warrant reporting. Improvement of care at primary level is important for both patient satisfaction and better health outcomes. [16,17] Such improvement requires provision of appropriate guidelines and a process to ensure continuing implementation of such guidelines.
An audit, defined as ‘an evaluation of a system, a process, a project, a product or an organization’, is an invaluable tool for this purpose. Many primary care practices do run occasional audits in an effort to selfassess and benchmark performance. Unfortunately, just as ‘one swallow does not make summer’, a single audit often does not crystallise a clear picture of reality. It does not assist in proper orientation and alignment of practice as it is supposed to do.Apractice is a maze of complex strategic and tactical decisions. Moving through such labyrinthine diversity, like driving in a busy city centre, one occasional look at a signboard may not be a good warranty for being on track; one needs an ongoing monitoring of the path and direction.
There does not exist, in most primary care facilities, any system of regular application, analysis and followup of audit data incorporating random variation in ongoing evaluation of services. Even where the process of assessment and review in implementing quality exists, it is often subjective and without explicit reference to predetermined standards of practice.[18]
The existence of natural, random variation in all processes makes the analysis and interpretation of the results of such measurement difficult unless methods are used to take into account such variation.[19] Such techniques, which are important for monitoring clinical practice,[20] include statistical process control, which, although most commonly applied in manufacturing industries, also has its application in service industries, like health care.[21] Use of this technique assists in detecting variations in performance due to avoidable causes and adaptation of service to reduce such variation,[22] providing a mechanism for continuous, concurrent evaluation and improvement.[23]
The findings of this study are important and relevant to clinical practice. For care provider behaviour modification we have shown the use of continuing process audits in the framework of statistical process control. The whole initiative can be maintained by the existing workforce and within the time constraints of a busy practice. Through ensuring an optimum process of care, one hopes to achieve better clinical outcomes in the long run.
In retrospect, we felt that once or twice a month process audits, coupled with once or twice a year outcome audits, would be ideal for inducing and maintaining any practice change. We intend to implement such an initiative in our practice, and results will be reported in future. There are many examples of audit in the literature between 1995 and 2005,[24–30] but all of them are about a single audit, except one study comparing two audits.[31]
We have presented the results of two series of audits in this paper, comparing their effectiveness in producing the desired result, that of minimising variability in care provision.
The staff of King Saud City Family and Community Medicine Centre for their dedication and hard work in pursuit of a higher purpose.
The study was conducted utilising the available resources in the primary care facility concerned.
Permission for running audits was sought from the hospital authorities. No human intervention was involved.
None. The author has no pecuniary interests outside the parent organisation, Saudi Arabian National Guard Health Affairs. There have been no involvements that might raise the question of bias in the work reported or in the conclusions, implications or opinions stated.
