Quality in Primary Care Open Access

Keywords

Screening; Primary care; Alcohol use; Veterans depression

Introduction

It is recommended that primary care serve as a platform for secondary prevention efforts to help increase prompt recognition and treatment of hazardous alcohol use and depression. These recommendations stem from a variety of factors that include the prevalence of hazardous alcohol use and depression in primary care patients;1,2 the effects that they have on morbidity, mortality, and quality of life decrements;3-6 the existence of empirically-validated brief screening measures that can be used to identify symptoms;7,8 and the patient and healthcare savings of early detection and initiation of evidence-based interventions for depression9 and hazardous alcohol use.10

Screening is a systematic way to help primary care providers/ staff identify patients experiencing symptoms of hazardous alcohol use or depression. Therefore, systematic screening can help facilitate treatment planning for a large number of patients whose symptoms would otherwise go unrecognized and thus untreated.11-14 Despite their values to the quality of clinical care, screens tend to be utilized infrequently.15 While prior research has identified a number of individual motivational and attitudinal factors contributing to inadequate screening, there are a number of organizational factors that can affect screening implementation as well.16, 17

To this end, the Veterans Health Administration (VHA) represents an excellent example of a large healthcare system that has successfully implemented standardized annual screening for hazardous alcohol use in 200418 and depression in 2008.19 These implementations involved the mandated use of the Alcohol Use Disorders Test-Consumption (AUDIT-C)8 and the Patient Health Questionnaire-2 (PHQ-2)7,20 as the screening measures in a larger implementation of clinical practice guidelines for Substance Use Disorders and Major Depressive Disorders.9,10 The implementations were supported by the integration of behavioral health in primary care and the creation of standardized electronic clinical reminders that functioned as a way to notify primary care staff when a screen is due as well as for follow-up purposes.19,21 The use of these electronic clinical reminders has helped greatly in ensuring the implementation of the alcohol and depression screens, as well as guiding providers through the process of screening, follow-up assessment, and the general intervention process.22,23

Although given support, the guidelines did not describe how to incorporate behavioral health screening into clinical processes. Therefore, each VHA clinic decided who would administer the screens, the modality by which the screens would be administered (e.g., some VA primary care clinics have patients complete using tablets; Pomerantz et al., 2010),24 when the screens would be administered, and how training would occur for staff. This freedom in implementation allowed each primary care clinic to identify the best way for their clinic to implement the screens while balancing the need to continue to maintain a high patient flow and trying to identify high-risk patients. This flexibility is supported by research on innovation assimilation, such that “lack of formalization” or flexibility around the use of an innovation is a factor uniquely associated with the adoption of innovations in service organizations.25

This institutional flexibility combined with electronic and behavioral health support likely facilitated rapid assimilation and use of these screens within the VHA as recent data show that 93% of patients were annually screened using AUDIT-C.18 However, it may not be without cost. The AUDIT-C and PHQ-2 were initially designed as self-report paper questionnaires that were found to have high levels of reliability, sensitivity, and specificity for symptoms of the target disorders (i.e., alcohol use and major depressive disorders). While studies have shown that they can also be reliably administered using other procedures, such as verbally or electronically,26,27 this is dependent on following certain administration guidelines (e.g., reading items/responses verbatim, providing confidential space for administration), as research has shown even slight deviations can result in inaccurate or meaningless results.28 Additionally, flexibility around training may lead to an increase in such deviations. Consistent with this, research suggests there may be significant discrepancies between the results of AUDIT-C screens conducted within VHA clinical and research/program evaluation settings.29-31 Such studies suggest that flexible implementation of the AUDIT-C within clinical settings may not be as sensitive or specific as compared to implementation in research settings, where best practices for administration procedures are more likely to be followed to optimize data collection. To date, it is unknown whether these differences are also occurring for the PHQ-2 screens.

As non-VHA primary care clinics and healthcare systems contemplate the incorporation of behavioral health screens into routine practice, it is important to develop implementation plans, including quality monitoring processes that balance ease of adoption while maintaining administration quality to ensure the information is clinically useful. Therefore, the purpose of this study was to describe the preferred methods for implementation in an effort to serve as a guide for non-VHA primary care clinics to help improve quality monitoring processes in the future. Therefore, we specifically examined the prevalence of different administration methods and processes as well as training procedures at various primary care clinics within the VHA as reported by primary care staff as well as their personal opinions regarding alternate modes of administration within their clinic. In order to ask primary care staff concrete questions about screening practices while minimizing survey time, questions typically referred to the AUDIT-C due to the history of previous research identifying discrepancies in results. However, additional follow-up questions were also asked about the PHQ-2.

Methods

Setting

The VHA has more than 900 primary care clinics within 21 VHA regional networks across the United States employing all types of primary care staff, who were eligible to participate in this study.

Recruitment

Over the course of a 6 week period, we used the global VHA internal e-mail system to email a description of the study to a random selection of 3,932 primary care providers and staff belonging to one of 641 primary care listservs. Stratification techniques based on the VHA’s 21 regional network system were used to help ensure a representative national sample. There were no participant exclusion criteria. Each identified staff member received one recruitment email every two weeks over the 6 week period. The email included a description of the project and a hyperlink to access the web-portal to the anonymous web-based questionnaire. The study was approved by the Syracuse VHA Institutional Review Board.

Participants

A total of 1196 respondents began the survey, 1120 completed at least 40% of it (i.e., responding to the demographic items and general administration questions), garnering a 29% response rate. Of the 1120 respondents, a subsample of 484 participants who regularly (i.e., at least several times per week) administered the AUDIT-C and PHQ-2 were asked additional questions about screening practices and were the primary focus of this study. Both the full (n-1120) and subsample (n=484) had participants representing all 21 VISNs.

Measures

Interested employees were asked to log onto to a survey website (surveymonkey.com) to complete the questionnaire that was designed by the authors using item response theory as a framework for understanding proper administration practices and literature on primary care processes and common barriers.32-34 Respondents were asked to enter basic demographic information (Table 1) and complete several multiple choice questions about their perceptions of AUDIT-C administration in their clinic and training experiences (Table 2). For those who reported receiving training, additional multiple choice items asked more specifically who conducted the training, format, and the components of the training. Additional items were included to ask all of the respondents whether the administration practices and training experiences were the same for the PHQ-2 using dichotomous response options (yes/no). Depending on whether they indicated their clinic as primarily administering the AUDIT-C verbally (i.e., defined to the respondents as “VA staff reads the questions to the patient and fills in the response”), electronically (i.e., defined to the respondents as “patient fills out the AUDIT-C on a touchpad screen or directly on to an individual computer or staff turns screen to patient and have patient choose the answers”), or on paper (i.e., defined to the respondents as “VA staff gives the patient a questionnaire and patient fills it out on own) , respondents were then directed to separate pages of questions asking them about the perceived advantage of the administration style within their clinic and the perceived barriers to the other two formats rather than all formats to help reduce response burden (Table 3).

Table 1: Characteristics of full sample and subsample who regularly administer behavioral health screens.

Table 2: Typical administration styles in your clinic.

Table 3: The percent of individuals endorsing each advantages/obstacles of each administration style.

Statistical Analysis

Descriptive statistics (means, standard deviations, and percentages) were the primary way used to characterize patterns in responses. In order to compare distributions of responses in the subsample of 484 to the overall sample (n=1120), the chisquared test for goodness-of-fit was used with the proportions from the overall sample being used to derive “expected” values and the distributions in the subsample serving as the “observed” values. Regarding the advantages and disadvantages of various modalities, broader categories were created in order to help interpret the results (Table 3). Additional chi-square analyses were conducted to examine differences across specific comparable item responses regarding advantages and disadvantages of the modalities. All analyses were performed using IBM SPSS version 19.0.

Results

Respondents who did not complete at least 40% of the survey (n=76) were excluded. These individuals did not differ from included respondents in their professional role, age, length of time working in the VHA, or whether they regularly administered mental health screens to patients. However, chisquared analyses showed that respondents who did not complete the survey were less likely to work full-time in primary care (71.0%; p<0.001) than the 1120 respondents in the final sample. Descriptive information for the overall sample as well as the subsample of staff (n=484) who indicated that they regularly administer both mental health screens as part of their primary care duties can be found in Table 1. This subsample was significantly more likely to work in primary care full-time as nurses or primary care providers (PCPs), and was more likely to work in community-based outpatient clinics than the overall sample.

Current practices for administering the AUDIT-C are presented in Table 2. Respondents indicated that the AUDIT-C is primarily (>50% endorsement) administered verbally by nurses in the exam room. There was somewhat more variation in terms of the timing of the AUDIT-C administration during the primary care encounter, with the largest portion of respondents indicating that the screen is administered by the nurse prior to the PCP seeing the patient. The AUDIT-C was least frequently completed using paper questionnaires.

A large majority of the total respondents (81.3%) reported that the same individuals who administer the AUDIT-C also typically complete the PHQ-2 screen within their clinic. Similarly, a large portion of respondents (77.9%) also reported that the PHQ-2 is completed using the same method (i.e., administration style, location, timing) as the AUDIT-C screen.

Fewer than half of 484 respondents who regularly administered screens endorsed being trained in the AUDIT-C (41.1%) and PHQ-2 (49.0%). When the 199 respondents who were trained in the AUDIT-C were asked about the nature of their training, a majority of them (59.3%) described their training as occurring in an individual format and as being conducted by a nurse or nurse manager (70.1%). These trainings almost always included discussions of the administrative aspects of the AUDIT-C, such as how to enter the AUDIT-C into the electronic medical record (89.4%), the purpose of the AUDIT-C screen (87.9%), and what to do if the patient screened positive (80.9%). However, fewer than half of the respondents reported that their trainings included discussions of best practices, which include topics such as how to further assess alcohol use (46.7%; n=93), the importance of reading the AUDIT-C items verbatim (41.2%; n=82), and methods to increase patients’ comfort completing the questionnaire (30.2%; n=60).

Table 3 illustrates the advantages the respondents identified for the method of administration that was regularly used within their clinic and their perceived disadvantages of the other two methods. For those who used verbal administration, the items assessing the ease of administration and patient experience were the most endorsed advantages whereas the items assessing the efficiency of the method were endorsed the least. This is in contrast to those using electronic administration, who endorsed items representing the ease of administration and efficiency as clear advantages. However, the items assessing the patient’s experience of the administration method was endorsed the least by those using electronic administration as compared to paper and verbal methods. Regarding disadvantages, respondents using verbal methods endorsed negative patient experiences (e.g. seeing or reading material, privacy) and logistical obstacles as the most endorsed disadvantages to the use of paper and electronic methods of administration.

Discussion

VHA primary care clinics have had several years to refine their local procedures for the administration of behavioral health screens for depression and alcohol use, finding ways to balance the inclusion of these screens and the competing demands of maintaining high patient flow in primary care. Our goal was to describe the most frequent implementation and training procedures used within these practices. We also examined the opinions of staff on various modalities of administration in an effort to help guide future implementation efforts as well as provide information on potential methods to help increase the likelihood of accurate screening information. As other research has found evidence for potential quality issues concerning the screening information collected in VHA clinical settings,29-31 it was hoped that this implementation information can be used to help guide future quality monitoring efforts within VHA and outside of VHA to help ensure that the screening information collected in primary care is accurate.

Our data highlight several notable patterns regarding the process of screening within VHA primary care practices that appear to apply to both depression and alcohol screening measures. Verbal administration of these screens is most popular, and nurses generally appear to administer the screens when collecting vital signs or before the PCP comes into the exam room. Verbal administration seems favored because it is simple, quick, and does not require a questionnaire as the questions are contained within the electronic clinical reminder in the patient’s medical record, which helps to ensure the efficient flow of patients through the clinic. Conversely, paper or electronic administration may significantly hinder patient flow or increase feelings of staff inconvenience due to logistical concerns regarding access to a computer/questionnaire as well as the impact on patient experience.

One concern with the popularity of verbal administration of these types of self-report screening measures is the need to follow a greater number of prescribed best practices administration procedures (e.g. reading directions, items, and responses verbatim; defining a standard drink). Our data suggest that a significant portion of primary care providers/staff do not remember any training devoted to these screens within the VHA. If staff did remember receiving training, it often was “on the job” by a colleague orienting staff to common work procedures, but it often was not comprehensive, ignoring best practices for screening measure administration. Although the questions and responses are listed within the electronic reminder, this lack of training and knowledge among primary care staff may be the reason for the discrepancies between clinically administered AUDIT-C screens and those administered as part of research/program evaluation protocols.30,31 It would be important to investigate in future studies whether this lack of knowledge among primary care staff accounts for the increased number of discrepancies that have been observed in past research. As other research has shown that even minor changes to administration methods significantly affect screening measure sensitivity and specificity rates,28 it is an area that implementation and quality monitoring plans should consider to help ensure the success of secondary prevention efforts.

Although our data provide a glimpse as to the preferred procedures among a widely diverse set of primary care settings ranging from small clinics serving rural populations to larger clinics in medical centers, there are also several limitations of this study. All data were self-report, and although there is no a priori reason to believe that primary care staff had any incentive to misrepresent their clinics’ practices, the accuracy of the data cannot be confirmed. In addition, our method for identifying primary care staff relied on email listservs and yielded a 29% response rate which is slightly lower than other studies using email recruitment methods (meta-analysis found typical response rate was 33%)35 causing some caution to the generalizability of our results. However, the diversity in the staff, the large number sampled, and the representation from all VISNs improves the strength of the findings. In addition, only staff who worked in VHA primary care clinics was surveyed. As these clinics do have to follow a specific mandate to screen and use specific screening measures that are supported by the electronic medical record, as well as have at least one integrated behavioral health provider to assist when a patient screens positive, there may be some aspects of the data that may not translate to non-VHA primary care settings. However, the VHA struggles with the same challenges as the private sector when it comes to patient flow and the need to identify high-risk patients. Therefore, it is believed these findings can help inform efforts to implement screening in varied contexts.

Implications

This study presents a glimpse into the challenges that emerge when attempting to implement behavioral health screening at a wide scale. It is likely that the impact of each administration style on patient flow and patient comfort likely play a similar role in determining the way the screens are implemented in many other healthcare settings as they did within the VHA. However, conducting similar studies in those settings may provide further insight into a given hospital’s own unique challenges.

Furthermore, recognizing the gap between best practices and actual implementation highlights the need for more formalized training to close these gaps. Constructing an informational training that might be included in an implementation plan targeting how to administer these screening measures is a highly feasible solution to ensure proper administration methods and may result in increased utility of the screening data that are collected. Past research indicates that similar “how to” informational trainings targeting other simple skillsets (e.g., neurological exam) can improve administration consistently.36 Another way to potentially reduce administration errors is to consider creating a general administration process for the clinic that involves administering the screens via electronic or paper means. It would be useful in future work to assess patients’ comfort with electronic or paper systems and how to maintain confidentiality as they respond to the questions. Once an implementation plan is identified, then continued attention to monitoring the quality of administration practices among primary care staff and continued improvement processes are also important for managers to identify to ensure the information collected is useful within clinical practice.

Acknowledgments

The views expressed in this article are those of the authors and do not reflect the official policy of the Department of Veterans Affairs or other departments of the United States government. This material is based upon funding from the VA Substance Use Disorders Quality Enhancement Research Initiative Locally Initiated Project

References

1. Hankin CS, Spiro A, Miller DR, Kazis L. Mental disorders and mental health treatment among U.S. Department of Veterans Affairs outpatients: the Veterans Health Study. Am J Psychiatry 1999; 156: 1924-1930.
2. Hasin DS, Goodwin RD, Stinson FS, Grant BF. Epidemiology of major depressive disorder: results from the National Epidemiologic Survey on Alcoholism and Related Conditions. Arch Gen Psychiatry 2005; 62: 1097-1106.
3. Cuijpers P, Smit F. Excess mortality in depression. J Affect Disord 2002; 72: 227-236.
4. Spitzer RL, Kroenke K, Linzer M, Hahn SR, Williams JB, et al. Health-related quality of life in primary care patients with mental disorders. JAMA 1995; 274: 1511-1517.
5. Broadhead WE, Blazer DG, George LK, Tse CK. Depression, disability days, and days lost from work in a prospective epidemiologic survey. JAMA 1990; 264: 2524-2528.
6. Room R, Babor T, Rehm J. Alcohol and public health. Lancet 2005; 365: 519-530.
7. Kroenke K, Spitzer RL, Williams JB. The patient health questionnaire-2: Validity of a two-item depression screener. Medical Care 2003; 41: 1284-1292.
8. Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption--II. Addiction 1993; 88: 791-804.
9. Department of Veterans Affairs & Department of Defense. VA/DoD Clinical Practice Guideline for the Management of Major Depressive Disorder 2009.
10. Department of Veterans Affairs & Department of Defense. VA/DoD Clinical Practice Guideline for the Management of Substance Use Disorders 2009.
11. Cepoiu M, McCusker J, Cole MG, Sewitch M, Belzile E, et al. Recognition of depression by non-psychiatric physicians--a systematic literature review and meta-analysis. J Gen Intern Med 2008; 23: 25-36.
12. Pignone MP, Gaynes BN, Rushton J, Burchell CM, Orleans CT, et al. Screening for depression in adults: a summary of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med 2002; 136: 765-776.
13. Reinholdz H, Fornazar R, Bendtsen P, Spak F. Comparison of systematic versus targeted screening for detection of risky drinking in primary care. Alcohol Alcohol 2013; 48: 172-179.
14. Vinson DC, Turner BJ, Manning BK, Galliber JM. Clinician suspicion of an alcohol problem: an observational study from the AAFP National Research Network. Ann Fam Med. 2013; 11: 53-59.
15. Rodriguez HP, Glenn BA, Olmos TT, Krist AH, Shimada SL, et al. Real-world implementation and outcomes of health behavior and mental health assessment. J Am Board Fam Med. 2014; 27: 356-366.
16. Johansson K, Bendtsen P, Åkerlind I. Early intervention for problem drinkers: readiness to participate among general practitioners and nurses in Swedish primary health care. Alcohol and Alcoholism. 2002; 37: 38-42.
17. Kaner EF, Heather N, McAvoy BR, Lock CA, Gilvarry E. Intervention for excessive alcohol consumption in primary health care: attitudes and practices of English general practitioners. Alcohol Alcohol 1999; 34: 559-566.
18. Bradley K, Williams E, Achtmeyer C, Volpp B, Collins BJ, et al. Implementation of evidence-based alcohol screening in the Veterans Health Administration. Am J Manag Care 2006; 12: 597-606.
19. Department of Veterans Affairs. Clinical Reminders 2.0 Patch 6 User Manual 2007.
20. Arroll B, Goodyear-Smith F, Crengle S, Gunn J, Kerse N, et al. Validation of PHQ-2 and PHQ-9 to screen for major depression in the primary care population. Ann Fam Med 2010; 8: 348-53.
21. Department of Veteran Affairs. Uniform mental health services in VA medical centers and clinics. VA Handbook 2008.
22. Williams EC, Lapham G, Achtmeyer CE, Volpp B, Kivlahan DR, et al. Use of an electronic clinical reminder for brief alcohol counseling is associated with resolution of unhealthy alcohol use at follow-up screening. J Gen Intern Med. 2010; 25: 11-17.
23. Lapham GT, Achtmeyer CE, Williams EC, Hawkins EJ, Kivlahan DR, et al. Increased documented brief alcohol interventions with a performance measure and electronic decision support. Med Care 2012; 50: 179-187.
24. Pomerantz AS, Shiner B, Watts BV, Detzer MJ, Kutter C, et al. The White River model of collocated collaborative care: A platform for mental and behavioral health care in the medical home. Families, Systems, & Health 2010; 28: 114-129.
25. Damanpour F. Organizational innovation: A meta-analysis of effects of determinants and moderators. Academy of Management Journal. 1991; 34: 555-590.
26. Pinto-Meza A, Serrano-Blanco A, Penarrubia MT. Assessing depression in primary care with the PHQ-9: can it be carried out over the telephone? J Gen Intern Med 2005; 20: 738-742.
27. Rumpf JJ, Hapke U, Meyer C, John U. Screening for alcohol use disorders and at-risk drinking in the general population: psychometric properties of three questionnaires. Alcohol Alcohol 2002; 37: 261-268.
28. Broyles LM, Gordon AJ, Sereika S, Ryan C, Erlen JA. Do words matter? Incongruent responses to inconsistently worded AUDIT-C alcohol screening instruments. Subst Abus 2011; 32: 202-209.
29. Hawkins EJ, Kivlahan DR, Williams EC, Wright SM, Craig T, et al. Examining quality issues in alcohol misuse screening. Substance Abuse 2007; 28: 53-65.
30. Krenek M, Maisto SA, Funderburk JS, Drayer R. Severity of alcohol problems and readiness to change alcohol use in primary care. Addict Behav 2011; 36: 512-515.
31. Bradley KA, Lapham GT, Hawkins EJ, Achtmeyer CE, Williams EC, et al. Quality concerns with routine alcohol screening in VA clinical settings. J Gen Intern Med 2011; 26: 299-306.
32. Embretson SE, Reise SP. Item response theory for psychologists. New York: Taylor & Francis; 2000.
33. Krist AH, Phillips SM, Sabo RT, Balasubramanian BA, Heurtin-Roberts S, et al. Adoption, reach, implementation, and maintenance of a behavioral and mental health assessment in primary care. Annals of Family Medicine. 2014; 12: 525-533.
34.  Rahm AK, Boggs JM, Martin C, Price DW, Beck A, et al. Facilitators and barriers to implementing SBIRT in primary care in integrated health care settings. Substance Abuse 2014.
35. Shih T, Fan X. Comparing response rates in e-mail and paper surveys: A meta-analysis. Educational Research Review 2009; 4: 26-40.
36. Stecker MM, Stecker MM. The effect of education on nurses’ assessments in an epilepsy monitoring unit. Can J Neurosci Nurs. 2012; 34: 23-32.