Research Paper - (2014) Volume 22, Issue 1
1National Institute for Health Research, Nottingham, UK
3Professor of Primary and Prehospital Health Care Community and Health Research Unit (CaHRU), University of Lincoln, UK
Received date: 20 December 2013; Accepted date: 6 January 2014
Background Treatment fidelity has previously been defined as the degree to which a treatment or intervention is delivered to participants as intended. Underreporting of fidelity in primary care randomized controlled trials (RCTs) of complex interventions reduces our confidence that findings are due to the treatment or intervention being investigated, rather than unknown confounders. Aim Weaimed to investigate treatment fidelity (for the purpose of this paper, hereafter referred to as intervention fidelity), of an educational intervention delivered to general practice teams and designed to improve the primary care management of insomnia. Method We conducted telephone interviews with patients and practitioners participating in the intervention arm of the trial to explore trial fidelity. Qualitative analysis was undertaken using constant comparison and a priori themes (categories): ‘adherence to the delivery of the intervention’, ‘patients received and understood intervention’ and ‘patient enactment’. Results If the intervention protocol was not adhered to by the practitioner then patient receipt, understanding and enactment levels were reduced. Recruitment difficulties in terms of the gap between initially being recruited into the study and attending an intervention consultation also reduced the effectiveness of the intervention. Patient attributes such as motivation to learn and engage contributed to nthe success of the uptake of the intervention. Conclusion Qualitative methods using brief telephone interviews are an effective way of collecting the depth of data required to assess intervention fidelity. Intervention fidelity monitoring should be an important element of definitive trial design.
fidelity, general practice, insomnia, primary care, qualitative, randomised controlled trial
Effectiveness trials in primary care aim to determine whether an intervention works in an experimental setting. For internal validity and effectiveness to be correctly attributed to the intervention being tested, it is important that intervention fidelity is maintained. Intervention fidelity is also important for external validity so that results can be generalised and treatments translated into everyday practice when delivered by typical primary care general practitioners (GPs) and practice nurses.
With the current emphasis on more health problems being managed in primary care, it is increasingly important to establish the most effective way to achieve positive health outcomes in this setting.[1] In many cases, this means that new approaches are required and primary care practitioners need to adopt and apply consistently techniques that have previously not been part of their regular practice.[2] Such changes need to be based on the best possible evidence and hence randomised controlled trials (RCTs) to test the effectiveness of the new treatments or interventions are a high priority. To reflect this need there has been a substantial increase in the number of RCTs undertaken in primary care over recent years.[3] Because many of the new practices need to be introduced to practitioners before they can be delivered, the interventions are often complex and educational in nature.[4]
Complex interventions hypothesised to improve a medical condition or service are, like other interventions, tested for efficacy by determining if any differences between the intervention and non-intervention group are clinically important and statistically greater than would be found due to chance alone. However, by the nature of such studies, whether significant differences are found or not, the researcher cannot be sure of the extent to which the intervention was delivered exactly as the researcher had intended. Neither can they be sure of whether the patient or primary care practitioner received or used the intervention as planned.
Treatment fidelity has been defined as the degree to which a treatment or intervention is delivered to participants as intended.[5] Treatment fidelity, hereafter referred to for the purpose of this paper as intervention fidelity, is a crucial measure for the accurate interpretation of research data. However there is little published literature around intervention fidelity in studies of complex educational interventions in primary care.
Monitoring intervention fidelity during the pilot phase of a study provides information that can explain study outcomes and be used to improve the design for a definitive trial with regards to several aspects of the delivery of the intervention. With strengthened internal and external validity in this way, the opportunity to maximise clinical changes due to the intervention are increased.
Bellg et al6 identified five areas for behavioural health researchers to consider when identifying intervention fidelity in their studies. These include:
• ‘design of study’ – ‘ensuring that a treatment can adequately test its hypotheses in relation to its underlying theory and clinical processes’;
• ‘training providers’ – ‘assessing the training of treatment providers to ensure that they have been satisfactorily trained to deliver the intervention to study participants’;
• ‘delivery of treatment’ – ‘using procedures to standardize delivery and checking for protocol adherence’;
• ‘receipt of treatment’ – ‘monitor and improve the ability of patients to understand and perform treatment-related behavioural skills and cognitive strategies during treatment delivery’; and
• ‘enactment of treatment skills’ – ‘monitor and improve the ability of patients to perform treatment- related behavioural skills and cognitive strategies in relevant real-life settings’.
We aimed to investigate intervention fidelity as one aspect of the pilot study of an effectiveness intervention, as opposed to a clinical treatment, using qualitative methods to evaluate three of these dimensions of fidelity, namely delivery of the intervention, receipt of treatment and enactment of treatment.[7]
This was a qualitative study embedded in a pilot cluster RCT which was conducted to test procedures and collect information in preparation for a larger definitive cluster randomised trial. The aim of the pilot trial was to establish the effectiveness of an educational intervention for general practice teams to deliver problem-focused therapy for insomnia.[8] For our educational intervention to be effective, we were expecting GPs and nurses to change their behaviour towards the management of insomnia, as well as expecting behaviour changes in patients themselves to bring about improvements in outcomes.
We already knew from the literature how intervention fidelity could be compromised.[6] For example, whether the intervention is delivered in the specific way that it was intended could be influenced by practicalities such as the time available in consultations to discuss the intervention, but we did not understand why compromises to fidelity may have emerged in this specific pilot RCT.
The ‘why’ element could not be disentangled by employing quantitative methods – it was considered necessary to collect rich in-depth data from those directly involved in delivering or receiving the intervention. We used a qualitative approach in order to be able to describe potential breaches in fidelity that arose and gain an insight into why they were occurring. We therefore conducted short telephone interviews with patients and practitioners that participated in the pilot RCT to monitor intervention fidelity.
Patients were invited to participate in a short telephone interview midway through the trial to discuss their recent consultations for insomnia with their GP/ practice nurse. They were invited by letter to participate and if they wanted to contribute, they were asked to return a slip in the reply paid envelope provided.
Practitioners were also invited to share their experiences of utilising the intervention in mandatory telephone interviews as part of a commitment to participation in the pilot RCT trial.Ten interviews were conducted by one of the research team (FT) each lasting approximately 10– 20 minutes. The interviews took place approximately halfway through the pilot RCT follow-up period in March 2010. Interview recordings were transcribed verbatim by the same researcher (FT) and data were analysed by two researchers (FT and JD).
We employed template analysis[9] using a set of a priori themes (categories) that aligned with the aspects of fidelity that we wanted to investigate: adherence to delivery of the intervention, patient received and understood intervention, and patient enactment. Data were coded by two researchers, JD and FT, to identify barriers and facilitators to these components of intervention fidelity as a means of understanding why any breaches in fidelity were occurring in these domains. After initial coding, the themes were reconsidered for overlap and contradiction. Additional de novo themes were developed as part of the iterative process. Themes were interpreted in the context of informing improved intervention fidelity and fidelity monitoring from a combined patient and professional perspective. Data were managed using MaxQDA.[10] To assist the reader with understanding the information given to us in the interviews, we have outlined the detail of the intervention (CBTi) in Table 1.
Table 1: Planned intervention process for pilot cluster RCT
There were six patient responders (Pt) and four practitioner responders (Pr). The cost of the fidelity study was £1286.33. A summary of the coding is presented in Table 2.
Table 2: Intervention fidelity categories and themes
Our findings indicated that practitioners perceived that they were delivering the intervention according to the protocol in terms of both process and content; they told us this in detail. However, patients contradicted this. When the practitioner did not adhere to the protocol, patients were confused, which led to them not really understanding what the research was about.
In more than one instance practitioners failed to hand out the sleep diaries at the first recruitment consultation as required by the protocol, leading to a lack of engagement with the patient. Even when they did, the patient was sometimes left uncertain about the purpose of the next consultation, the first intervention consultation.
Try to talk them into the idea of the sleep diaries and establishing exactly what the pattern of their sleeping was. (Pr 4) The practice nurse just gave me, basically, the sheets what I’m to fill in for my, this other, for my diaries and just went through the sheets that I had to fill in but then nothing else really. (Pt 1)In some cases, the intervention was not completed because the practitioner failed to make it clear that a follow-up appointment was vital as part of the intervention.
I’m not, as you may have gathered, the sort of person that frequently goes to the doctors. They would expect me to handle things on my own and go when there was something specific to report or ask about. (Pt 6)
Despite a detailed recruitment plan being given to the practitioners, there was evidence that suitable patients were not being recruited, i.e. patients that wanted to improve the quality of their sleep.
Well quite honestly I’ve accepted my sleep pattern and I’m happy with it. (Pt 2)
Well I’ve seen two, neither of them have actually come back because as I say one followed it up by saying, to be fair to her she was quite elderly, and then the next one basically rang to say the problem wasn’t a problem anymore. (Pr 4)
This was further exacerbated by the fact that the timing between the training for the practitioners and the recruitment was too great for both practitioners and patients and the intervention lost momentum due to too long a gap between appointments.
•.. think the biggest problem was the kind of the lack of momentum in terms of, I think if we’d got cracking when we’d all had the training at the start I think that would have made a hell of a difference ’cause I think we would have all been working to the same timetable and all been on the same wavelength. (Pr 4)
I thought that it had all gone, you know, dead for some reason, money had been withdrawn, or I don’t know, I didn’t think any more about it. I’d almost forgotten about it actually. (Pt 5)
The data from two patient interviews suggested that the intervention was not delivered to plan because the practitioner was uncommitted and appeared not to want to take part or believe in the intervention.
I got the distinct impression that the lady I was dealing with thought the whole thing was a waste of time. (Pt 5)
I got the distinct impression that the lady I was dealing with thought the whole thing was a waste of time. (Pt 5)
When the practitioner evidently followed the intervention protocol, the patients were delivered the intervention as intended.
What time, you know, did I go to bed and, you know, what time did I go to sleep and how long did I stay asleep and when did I wake up and when did I have a problem sleeping. (Pt1)
Well they brought back the sleep diary, back to show me, so we went into it and discussed the pattern and I made sure I understood what they had put down and that they had completed it and I also asked them in quite a lot more detail about the sorts of things that they do in the day ... (Pr 2)
Well. I gave them some literature, we then went through what their perceptions of the options were ... and then really went over the sort of guidance about trying to stick to regular patterns, reduce caffeine ... try and do everything they can to promote sleep in terms of reading, relaxation etc. and then actually go on to the other issues like exercise and alcohol. (Pr 4)
Practitioner’s usual behaviour also played a part in facilitating the delivery of the intervention
It’s a normal part of my kind of repertoire in other parts of medicine where I use lots of patient information leaflets and self-help programmes so it fits in quite nicely with how I practice anyway. (Pr 1)
If the practitioner was unable to explain the study rationale and processes, the patient may not have understood their role within it. An example of this was where the patient confused the assessment for recruitment into the study with the assessment of their sleep to monitor progress with the intervention.
Oh the practice nurse just gave me the forms to fill in so really there wasn’t much conversation apart from filling in the forms on that score. (Pt 4)
The researchers expected practitioners to incorporate the intervention in their consultation using their previously learned communication skills to explain clearly what was required of the patient. Instead practitioners were sometimes overly focused on filling out the forms and paid insufficient attention to applying the assessment and treatment.
When I started I didn’t really know what it was going to be. I got the impression that whatever I was doing I was doing it for the benefit of, you know, the public at large. (Pt 5)
Right ... should I now, six weeks on, be looking at this pattern myself and saying well ought I be cutting down on the caffeine? (Pt 6)
The only contact that I’ve had of recent times was the practice nurse who just sort of, didn’t really ask me anything in particular apart from the fact of filling in the form. (Pt 4)
SUBTHEME: INTRINSIC TO PATIENT
Even when the practitioner delivered the intervention correctly, there were aspects of the patients’ attitude that enhanced their ability to understand and engage with the intervention; critically important was having an open mind without preconceived expectations of the treatment they would receive.
Most patients actually it seems that they don’t want to have sleeping pills, they’re quite happy to look at alternatives. (Pr 1)
Interviewer: When you went to see the GP did you want a prescription for sleeping tablets? I wanted to sleep. (Pt 1)
Patient motivation to learn and engage with the research was also an important attribute.
I’m so willing to try something else, I mean if they suggested that, maybe some more hypnosis or something apart from these pills would be wonderful. (Pt 4)
I’m so willing to try something else, I mean if they suggested that, maybe some more hypnosis or something apart from these pills would be wonderful. (Pt 4)
As well as being familiar with and keen to deliver the intervention as part of their responsibility and commitment to the research project, practitioners used explanations that were tailored to the patient to enhance their understanding.
So looking at the diary we agreed that she was getting up at the right time and that was fine and looking at her diary we agreed that she was drinking a lot of caffeinated drinks so we talked about reducing caffeine and reducing her food and drink intake nearer to bedtime. (Pr 3)
I’ve been doing these sleep diaries and she said ‘oh yeah ok then look at the caffeine’, ‘it’s not coffee its only tea’ and she said ‘it’s the same in tea as it is in coffee’... and so I said ‘well what about decaffeinated’ and she said ‘oh yeah’, well I can’t tell the difference when I’m drinking it, so I now keep some in my handbag and if I’m going anywhere I use my own tea bags (oh good) that has helped. (Pt 1)
Patients’ motivation to learn was enhanced when practitioners took on the responsibility to recruit people for whom sleep was a genuine problem and who wanted to resolve it.
I just wanted to sleep but I had depression as well so it’s all part of ... I mean my husband died five years ago and it’s, I really, well it’s just beginning to hit me now, well it was about a year ago, so that’s really where it all came from, so she said would I be interested in this sleep programme because she wanted to do it and she was looking for people who would listen and take notice of it. (Pt 1)
As might be expected, understanding their role in the research and the information they received played a major part in patients being able to understand and carry out the advice to change their behaviour. The process whereby practitioners delivered and patients adopted CBTi advice, we have called ‘enactment.’
So we talked about trying to stop work earlier and we identified the fact that basically whatever time you said to him to go to bed he would work up to five minutes past that time because that’s the sort of person he is, a cramming-it-all-in person as such and he recognised that. (Pr 2)
For some patients being in the research project was a motivator to adhering to the advice and finding a solution.
I mean I have to say I keep a copy of the first lot I did but I was away when, I took the second lot away and I didn’t have reference, I did them off the cuff and I filled in the present lot that are ready to be posted and I’ve not looked at it but I might just take a copy and then when I’ve posted it I can compare the two. (Pt 6)
Knowing they were working in partnership with the practitioner was clearly important to patients, not only for enactment of the content of the intervention, but also for maintaining engagement throughout the project.
But I amsleeping better than I was and we have changed a few things. (Pt 1)
We also, because she always does quite a lot of exercise so we talked about moving that to an earlier part of the day, using the hot bath and then straight to bed technique not watching TV and reading (yeah) and we also talked about using the ‘the’ word because she does find it really difficult to switch off. (Pr 3)
Practitioners, by providing explicit instructions of what was expected of both the patient and themselves, were often helpful in encouraging patients to put the intervention into practice.
So I sort of gave her about three or four action points and she is due to come back and see me again in another two weeks. (Pr 3)
So I was already on the sleeping tablets and we were, you know, keeping it going and then she said how would you like to try and sleep without them and I said that would be absolutely wonderful, you know, let’s do it. (Pt 1)
If the patient did not have a sleep problemthat caused them sufficient difficulty for them to be willing to make changes in their behaviour, then they were less likely to follow the advice given to them. Sometimes the screening questionnaire referred to other problems (e.g. PHQ9 for depression) which some patients found frustrating.
It isn’t really a problem, it’s a nuisance. (Pt 5)
Well no, no offence, but if it’s the same set of questionnaires there’s little point, nothing will have changed. I’m not depressed, I don’t want to commit suicide ... you know so it just seems a bit pointless to answer the same things for the fourth time. (Pr 2)
In some instances, lack of motivation to change behaviour was a barrier on its own.
He said you don’t think of getting up and going elsewhere in the house and I said I don’t because it’s not usually that long and [sighs] I know it’s not a big thing but I’ve got to cancel the alarm if I go somewhere else in the house and it’s just another thing, I think well I’ve shut the house up for the night I’m going to stay in my bedroom. (Pt 6)
Patients who had already found a solution to the problem were also unlikely to follow the research through to completion. This was also true if the patient did not have their expectations of the consultation met.
Now I have found a way of, well not dealing with that, but helping around it. (Pt 5)
He was the one that suggested that maybe I was expecting too much if I was looking for about six hours sleep, then I’d have to content myself with about four and I said ‘oh dear surely not because I already feel tired all the time’.
Patients who did understand the intervention and their role in the research might still have failed to enact their knowledge if they felt that the advice was not personalised to them, although this might also be the result of the practitioner relying too much on using patient information leaflets rather than explaining CBTi directly to the patient.
I mean I took everything away and read it all seriously and ... I thought I just don’t see how they are necessarily going to help me because so many of them to me seemed to be in the getting off to sleep stage. (Pt 6)
Our purpose for this qualitative assessment of intervention fidelity was to identify how we might improve from our pilot study8 in order to increase our confidence in any results obtained in the proposed definitive trial. Recommendations to this purpose are therefore the key outcome of this paper. Our results reveal instances where the intervention was delivered dependably to bring about the hypothesised changes in behaviour and consequently quality of sleep. This indicated the robustness of the intervention development. [8,11–13]
A weakness of our study was the small sample size. We interviewed only six patients and four practitioners and the generalisability of the findings may therefore be limited. However, the small sample size could alternatively be considered a strength, because clear findings emerged and themes were consistent throughout the interviews.
In each area of fidelity that we explored, namely adherence to delivery of intervention, receipt of intervention and enactment of intervention,6 there were instances in which fidelity was independently breached. There were also interrelationships between these areas in that lack of fidelity in adhering to the protocol influenced the likelihood that fidelity in the other two areas would be reduced, and similarly, if breakdown in fidelity occurred at the level of receiving the intervention then enactment could also be undermined.[14] The implication of this is that monitoring adherence to delivery needs to be built into the design of the definitive randomised controlled trial.
Problems with fidelity in this study were affected by issues of recruitment. As with all research, recruiting the right patients is critical, and this is even more so with a behaviour change intervention. To bring about behaviour change to overcome sleep problems using a problem-focused therapeutic approach required practitioners and patients to be at the ‘readiness to change’ stage in the behaviour change cycle.[15,16]
Our findings suggest that not all practitioners attending training in the intervention arm were ready to change their practice and that not all patients recruited were ready for the change in their behaviour to improve sleep. A key issue in this study was to ensure recruitment of patients for whom poor sleep was a significant problem, disrupting some aspect of their lives to an extent that they were motivated to work with the practitioner to bring about change – whatever that might be. Motivation was a key factor affecting fidelity with regards to receipt, understanding and enactment of the intervention for patients and delivering the expected intervention for practitioners. [17,18]
In line with our purpose, we have identified barriers to fidelity that need to be removed and facilitators which need to be actively encouraged. Our interpretation is that if the intervention delivery protocol, process and content (including recruitment) are adhered to as per training, the patient understands and is more motivated to enact.[19] Intervention fidelity was poor in a number of dimensions; despite standardised training, GPs and nurses did not deliver the intervention consistently and there was variation between practitioners. Non-specific intervention effects such as the level of involvement of practitioners and their interest in the patients had a high impact on patient involvement and hence fidelity of the intervention.[20]
Recommendations for improving the management of the definitive trial include introducing a mechanism for managing the intervention (recruitment of practices, GPs and patients and refresher training for GPs), monitoring delivery of the intervention in GPs and its receipt among patients, and supporting positive enactment of the intervention
Primary care practitioners selected to participate in the definitive RCT need to be motivated to deliver the intervention as originally designed. Similarly to patients, practitioners should not feel coerced into taking part, i.e. practice level involvement should be distinguished from individual practitioner participation. Other recommendations include: introducing a mechanism for monitoring participant complaints, providing regular updates for practitioners of the intervention with audiotapes of their sessions to provide feedback and maintaining a qualitative interview at the end of the study to discuss intervention progress.
Recruitment of patients was inconsistent and in some cases inappropriate, e.g. where patients were not motivated to change their current sleep patterns. We suggest that the responsibility for the recruitment of patients is transferred wherever possible from the primary care practitioners to a member of the research team. The researcher would conduct all of the recruitment consultations and monitor intervention appointments to ensure that these are not delayed. There should also be a mechanism for refresher training, for example delivered using e-learning.
Arguably, the most important component influencing intervention enactment in this pilot study was timing – too long a gap between practitioner recruitment training, patient recruitment, practitioner intervention training and intervention appointments reduced motivation and affected enactment fidelity. Amendments to the recruitment processes are desirable but would need to be carefully considered to avoid recruitment bias. In this pilot, practices were blinded during the period of patient recruitment to reduce recruitment bias. Participants recruited into complex intervention trials should not be aware of whether they will receive intervention or control treatments. A lack of blinding would reduce the internal validity of such a trial through differential recruitment. [21]
The general consensus obtained from analysis of the interview data suggested that there were too many data collection forms to complete and too much emphasis placed on the form completion rather than the therapy; this might be mitigated if better explanations were provided about what the various forms were for, leading to better understanding. Again, this could be remedied by introducing a researcher as the individual responsible for recruitment. They could spend more time talking to the patients during the recruitment consultation about the underpinning theoretical perspective that the forms are based upon and the application of the forms as a measurement of sleep quantity and quality.
To monitor adherence to the intervention protocol, practitioners could be asked to complete behavioural checklists after they conduct each intervention consultation. 6 Regular visits from a researcher might also be used to informally discuss and clarify any issues which may have arisen that could potentially affect progress of the intervention. In addition, the inclusion of computerised prompts and feedback for various steps of the intervention could be implemented to support the practitioners to fully adhere to the intervention protocol.[22]
Poor intervention fidelity with regard to whether the patients received and understood the intervention was closely linked with whether or not the protocol was adhered to; reinforcing the need for the recommendations for intervention delivery. The opportunity to monitor patient understanding of their role and the content of the intervention although present was being underutilised in the current protocol. We suggest that telephone interviews similar to those conducted as part of this evaluation are incorporated into the study design as a key method of assessing patients’ receipt and understanding of interventions delivered. For example, conducting telephone interviews with patients between day one and a week after their first intervention consultation would be particularly beneficial in monitoring fidelity – any discrepancies that emerged could be managed by the research team to ensure that the intervention is delivered as designed.
To foster a positive relationship between the practitioner and patient and a sense of ‘working together’ (which has previously been discussed to have implications for successful outcomes), it may be useful for practitioners to directly ask patients how they think that they could overcome any obstacles they have to changing their behaviours.[6] These could then be negotiated and refined as part of a team effort.
Formal methods of recording consultation objectives and goals could also be utilised.[6,23] For example, the practitioner could provide an appointment card at the end of each session detailing the next appointment and the techniques that had been jointly agreed that the patient would attempt during the time period between sessions.
In the original protocol,[8] practitioners were trained to recruit patients into the study but this was not effective as a method of timely recruitment because of conflicting priorities of clinician workload. As suggested above, it may be better for a researcher to be responsible for organising and conducting patient recruitment consultations. There are many reasons for this: recruitment of patients into the study would be the researcher’s main priority, whereas this is only one concern to practitioners; the researcher would have a clearer understanding of the type of patient that would be suitable for inclusion (i.e. a patient in the appropriate stage of the behaviour change cycle); and they would be able to dedicate more time and effort to the task. Consequently, the time between recruitment of patients into the trial and intervention consultations beginning should be shorter, enhancing the likelihood of enactment both in terms of the practitioner and patient perspective.
the designed intervention. We suggest that it would be beneficial to include similar fidelity-monitoring processes in definitive trials to improve the accuracy of findings and conclusions drawn. A small number of interviews are sufficient to produce useful and valid results. Overall, our findings demonstrate that qualitative methods with brief telephone interviews are a relatively inexpensive way of learning about intervention fidelity.
We thank patients and practitioners that participated in the research interviews and we thank the Peer Reviewers for their comments. This study was conducted as part of awider programme funded by the Health Foundation called Resources for Effective Sleep Treatment (REST), a quality improvement project which aims to implement psychological treatments for insomnia (www.restproject.org.uk).
JD was responsible for the data analysis and the writing of the first draft of the paper. FT was responsible for the recruitment of patients and practitioners into the intervention fidelity interviews and conducting the telephone interviews. She was involved in data analysis and revisions to the paper. NS was involved in the design of the study and revisions to the paper. All authors read and approved the final manuscript.
This work was funded by the Health Foundation.
This study was approved by the Derbyshire research ethics committee 08/H0401/89). Approval for research management and governance was sought and gained from NHS Lincolnshire.
Not commissioned; externally Peer Reviewed.
None.
