Acta Psychopathologica Open Access

Keywords

Expectancy; Homework; CBT outcomes; Change mechanisms

Introduction

Cognitive Behavioral Therapy (CBT) is the first-choice therapy for patients with panic disorder and agoraphobia [1-4]. Research results confirm that exposure is very important for treatment success [5-6]. Although CBT is recognized as an empirically welldocumented treatment, the exact mechanisms that cause a change have not been explored so far [7]. Studies investigating the effectiveness of CBT in psychotherapy have shown that the positive treatment expectations of a patient play a central role and are directly related to the therapy result [8-10]. However, again little is known about the specific mechanisms responsible for the relationship between treatment expectations and therapy outcome [11]. Westra et al. [1] found that the homework compliance of a patient with a panic disorder and agoraphobia fully mediated the relationship between positive treatment expectations and a positive change early in the course of the therapy.

The aim of the present study was to determine the role of pre-treatment expectations or compliance with homework influenced treatment outcome. Therefor we investigate whether the effects reported by Westra et al. [1] could be found in a different setting, while implementing improved methodology.

The importance of replication was demonstrated by a recently published international replication study by the “Open Science Collaboration” [12]. The research team emphasized that, although it is a scientific principle to discover new things, confirming existing findings is essential. Sample dependencies or systematic errors can lead to different results than in the original study. Replications thus not only make a valuable contribution to quality assurance, but also to the development of psychological theories and the generalization of findings [13].

Treatment expectations are the prognostic assumption of a patient about the consequences of a therapy. Usually treatment expectations are assessed in psychotherapy by asking about the potential gain or the expected negative effects of the therapy [14]. The positive treatment expectations of a patient lead to a higher perception of self-efficacy and this, in turn, has a positive effect on the patient’s motivation [15]. In a meta-analysis, Constantino et al. [14] reported a small (d=0.24) but significant positive effect of treatment expectations before therapy start on the treatment result, which means the more positive the expectations of a therapy are at the beginning, the more likely patients are to show an improvement at the end of treatment. This relationship appears to exist for various anxiety disorders (e.g. [8,10,16]). However the question arises as to which specific mechanisms cause the expectations of a patient to affect the change process [11]. Studies show that higher positive expectations of therapy lead to a greater degree of involvement, i.e., of participation in therapy [11,17,18], which in turn predicts better treatment results [19,20]. The involvement of a patient can be operationalized in various ways. One possibility is homework compliance. In a metaanalysis, Kazantzis, Whittington, and Dattilio (46 studies; n=1,072) [21] concluded that 62% of patients undergoing a therapy with homework improved significantly, whereas only 38% of patients undergoing a treatment without homework improved significantly. Mausbach, Moore, Roesch, Cardenas, and Patterson (23 studies; n=2,183) [22] reported similar results, showing a small to medium effect (r=0.26) of homework compliance on treatment outcome. Thus homework compliance can be viewed as a major process factor in treatment.

In a first attempt to investigate the specific mechanisms that influence the relationship between positive treatment expectations and treatment outcome and the link between homework compliance and symptoms at the end of the therapy, Westra et al. [1] conducted a study on a partial sample of 23 patients who suffered from a panic disorder with or without agoraphobia. The patients were treated in a total of 10 sessions with cognitive behavioral group therapy. The study confirmed the relationship between positive treatment expectations and a positive change early in the course of therapy. They also revealed that this relationship was no longer significant when homework compliance was added as a mediator into the model. High homework compliance at the beginning of treatment (measured after the second session) fully mediated the relationship. That is, positive treatment expectations alone did not seem to be the cause of positive early change. Patients had to participate in the therapy and complete their homework to experience a reduction in symptoms. These findings indicate that homework has a direct impact, improving treatment outcome. However, Westra et al. [1] demonstrated in their study that the direct influence of homework compliance on treatment results was no longer significant if change experienced by a patient early in the therapy was added as a mediator. The positive effects of CBT can usually be observed during the first sessions of treatment. Empirical studies have shown that “early response” to a treatment is an important predictor of therapy outcome [23-27].

Purpose of this Study

The aim of this empirical study was to explore the mechanisms of change in the therapy process of an outpatient CBT intervention for patients with panic disorder and agoraphobia. Replicability can be compromised by small samples, selective reporting, publication bias as well as low treatment integrity, and overestimation of small effects [28]. Therefore, we examined whether the findings of Westra et al. [1] could be generalized to a different setting in which patients were treated in individual therapies with a focus on in vivo exposure, instead of group therapy as implemented by Westra et al. [1]. Potential weaknesses were eliminated by increasing the effectiveness of the treatment and the statistical analyzes. This involved modifying and improving the methodology, using the mean value of sessions 1 – 5 as a measure of homework compliance (i.e., [29]) instead of capturing homework compliance after the second treatment session, as was the case in the study of Westra et al. [1]. Research is lacking on whether the time when homework compliance is measured plays an important role or not. There are only a few findings (e.g., [30-31]) which show that early homework compliance has a stronger relationship with treatment outcome compared to compliance in later sessions. If homework compliance at the beginning of treatment plays an important role, the mean value of the first treatment sessions is a more representative estimate than the value of a single session [32]. Furthermore, there is no evidence showing that this specific session is particularly important.

In addition, in this study, the German version of the panic and agoraphobia scale (PAS) [33] was used to measure the outcome. Westra et al. [1] captured early change and treatment outcome using the Anxiety Sensitivity Index (ASI) [34], which measures the extent to which the fear of physical changes is related to anxiety. However, this index only captures some aspects of panic and not of agoraphobia, failing to take into account that the disease is especially characterized by varying severity of panic attacks, avoidance behavior, restrictions in daily life, and health care. The PAS, as a well-established measurement, encompasses all areas that ultimately lead to impairment in the everyday life of the patient due to the disorder. Based on Westra et al. [1], we expected that a) the extent to which a patient complied with homework would mediate the relationship between positive treatment expectations before beginning the therapy and an early response in the CBT and b) that experiencing an early change would mediate the relationship between homework compliance and the symptoms at the end of the treatment.

Method

Study objectives and design

The study took place in the outpatient treatment center for psychological intervention at the University of Landau, Germany. After a telephone interview, in which the inclusion and exclusion criteria were verified, an initial face-to-face contact and structured clinical interview (SKID-I) [35] was carried out by a trained diagnostician who was not involved in the treatment. All psychotherapists had at least one year of professional experience, were trained to implement the therapy manual in a two-day workshop, and were supervised during therapy. After the survey of the baseline symptoms (pre-diagnostics – T1), the patients were treated according to the manual of Lang, Helbig- Lang, Westphal, Gloster, and Wittchen [36] in a total of 12 sessions, two sessions per week. The therapy therefore lasted a total period of six weeks. A study by Gloster et al. [7] showed the treatment manual to be highly effective. After each session, interim measurements were taken to record changes in symptom intensity.

Written informed consent was obtained from each patient and the study protocol conforms to the ethical guidelines of the World Medical Association (WMA) Declaration of Helsinki.

Participants

The sample consisted of patients who fulfilled the diagnostic criteria of the DSM-IV [37] for a panic disorder with agoraphobia as well as patients who had a total value of 19 or higher on the panic and agoraphobia scale [33]. People were excluded from the study if they had a previous history of bipolar disorder or psychosis or if they were currently suffering from bipolar or psychotic symptoms, suicidality, addiction or abuse of substance, an organically conditioned mental disorder, or currently undergoing a psychotherapy. Comorbidities with other anxiety disorders or a depression did not constitute exclusion criteria, as long as the panic disorder with agoraphobia was classified as the primary concern. Like in prior studies, if persons were taking psychotropic drugs or psychopharmaceuticals, they were not excluded if the drugs had been discontinued prior to starting the treatment or were taken constantly (beta blockers and benzodiazepines for at least one month, all other psychotropic drugs for more than three months) [38].

A total of 236 people contacted the university outpatient clinic by telephone. After verification of inclusion and exclusion criteria by telephone or personal contact, 196 persons were excluded from the sample. Thus, a total of 40 interested persons were invited to attend a diagnostic interview [35]. Re-examination of the inclusion and exclusion criteria resulted in the exclusion of another six persons. The remaining 34 patients met the study criteria and participated in the outpatient treatment for patients with a panic disorder and agoraphobia.

Completers were defined as patients who completed all 12 therapy sessions. Therapies that were not finished regularly were classified as drop-outs and were not included in the analyses. Twenty-three patients completed the treatment fully and were included in the data analysis; 73.9% of them were female. Table 1 shows the sample characteristics. The mean age was 36.9 years (SD=14.19). More than 50% of the sample reported suffering from anxiety disorder for more than two years. Only 39.1% were monosymptomatic; all others (60.9%) met the diagnostic criteria for at least one other mental disorder. The most common comorbid diagnosis was depression (39.1%). The characteristics of our sample were comparable with the sample characteristics in the study of Westra et al. [1] (N=23; age M=43.48, SD=9.45; 65% females).

Descriptive	N	%
Gender (N=23)
Female	17	73.9
Male	6	26.1
Age (N=23)
M	36.9
SD	14.2
Marital status (N=23)
Single	12	52.2
Married	9	39.1
Divorced	2	8.7
Education (N=23)
9 years of school	4	17.4
10 years of school	12	52.2
11 years of school	2	8.7
12 years of school	5	21.7
Chronicity of anxiety disorder (N=22)
3-6 months	2	9.1
6-12 months	3	13.6
1-2 years	5	22.7
more than 2 years	12	54.6
Comorbidity (N=23)
1 Major diagnosis	9	39.1
2 Diagnoses	4	17.4
3 Diagnoses	7	30.4
4 and more diagnoses	3	13.1
Comorbidity with depression (N=23)	9	39.1

Note: N=total; %=percentage; M=mean; SD=standard deviation.

Table 1: Sample characteristics.

Intervention

All patients were treated according to the therapy manual of Lang, Helbig-Lang, Westphal, Gloster, and Wittchen [36]. Homework was assigned at every session. The main focus of sessions 1 – 3 was on providing psychoeduaction about anxiety and developing motivation for the therapy. The aim of sessions 4 and 5 was, first, to explain the rationale behind interoceptive exposure and, in the next step, to perform the treatment accompanied by the therapist. In sessions 6 and 7, the patient was exposed to the first situations (e.g. public transport such as bus driving and visiting a department store), again accompanied by the therapist. Between sessions, patients had to practice on their own. Session 8 was used to reflect on past experiences during the exercises, to identify typical avoidance behavior and changes in the way of dealing with anxiety. In sessions 9 – 11, the patients were exposed to additional situations adapted to the patients’ individual fears. Finally, in session 12, conclusions were drawn based on the treatment together with the patient. The most important experiences were formulated, risk situations identified, and the learning experiences from the therapy were applied to these risk situations.

Measures

The independent predictor variable (positive treatment expectancy) and the dependent therapy success variable (severity of panic and agoraphobia measured with the PAS) were recorded at the beginning (measurement time 1 – T1) of therapy. The primary outcome criteria (severity of panic and agoraphobia, determined by the PAS and homework compliance) were also measured after each therapy session.

German version of the panic and agoraphobia scale (PAS)

Self-assessment questionnaire, consisting of a total of 13 items, which measures the severity of panic attacks, agoraphobic avoidance behavior, anticipatory anxiety, restrictions in daily life, and health care [33]. It has five different scales, for which separate totals can be calculated. The assessment period is the last seven days. Empirical findings confirm the retest reliability of the PAS; it has a sufficient internal consistency (r=0.78; α=0.85; [39]) and a high sensitivity to changes caused by the treatment [40].

Patient questionnaire on therapy expectation and evaluation (PATHEV)

Reliable and valid questionnaire, consisting of four items, for determining the extent to which the patient expects the imminent treatment to be successful [41]. Patients use a 5-point Likert scale, ranging from 0 “Not at all true” to 4 “True”, to evaluate their hope that the therapy will improve their health (e.g., “Even after therapy my problems will probably not change much”). The scale of “hope for improvement” yields good item characteristics and reliability (Cronbach’s α=0.89; in the present study even α=0.92).

Homework compliance

Patients assessed their homework compliance directly before each session for the period between the previous and the present session using a self-developed scale consisting of two items (“To what extent did you complete your homework?”, “How helpful was/were the task(s) in dealing with your problem?”). The answers to both questions were captured by a scale ranging from 1 “not at all” to 7 “very”. The values for the two items were averaged. This mean value was reliable in the present study (Cronbach’s α=0.87).

Definition of therapy success and early response

To measure the early response to therapy, the difference value of the PAS sum score from sessions 1 to 5 was used. For the treatment outcome the difference value of the PAS sum score from sessions 1 to 12 was used. The difference value as a criterion is suitable because the degree of change can be directly reflected and interpreted [42]. The reliability of such values has been wrongly criticized in the past (e.g., [43]). Indeed, they are preferable to other values [44]. By this modification, the interpretation changes in that a positive relationship is expected between the positive treatment expectation and the early change as well as between the homework compliance and the early change, because the higher the difference value, the greater the reduction in symptoms, that is, the lower the values for the PAS at the second measurement time.

The definition of early response was based on the definition used by Westra et al. [1]: the therapy session at which an individual’s score for the outcome measure had dropped by at least one full standard deviation in comparison to the baseline value.

Statistical Analyses

The analysis of data was performed using the program R [45]. To test the hypotheses, regression analyses were calculated. The classic procedure for testing a mediator model is usually based on the procedure of Baron and Kenny [46]. However research by MacKinnon, Lockwood, Hoffman, West, and Sheets [47], Shrout and Bolger [48], and Collins, Graham, and Flaherty [49] has revealed some difficulties with this approach. For this reason, in this study, we tested whether the indirect effect was significantly different from zero using a modern method, the so-called Monte- Carlo Method [50]. This method is considered to be the method of choice for verifying mediator hypotheses [51].

Due to the small sample size, a 95% confidence interval and a 95% bootstrapping confidence interval were produced when testing the significance of the regression analyses. That is, 5000 samples were repeatedly and randomly drawn from the data sample - thus, every person is put back in the sample immediately after having been drawn. As n-times draws with putting back leads to another person being drawn each time, this method provides slightly different values when the sample is drawn repeatedly. For reasons of replicability, therefore, the set. Seed command was used to ensure that repeated implementation resulted in the same values. The result of the regression analysis is significant when the lower and upper limits of the 95% confidence interval do not include zero [52].

Results

Early response

Early response according to the definition applied in this study was observed in patients (N=19) on average in session 5.11 (SD=2.94). Eleven of the 19 patients who experienced an early change showed it before or in the fifth treatment session. The remaining 12 experienced it after session five. For persons who had no reduction of baseline symptoms by at least one full standard deviation (N=4) – according to the definition of Westra et al. [1] – based on the calculated mean in the sample, the total value of the PAS from the 5^th session was used for the data analysis which included the early change.

Outcome evaluation

In our study, the cognitive behavioral therapy showed high effects according to Cohen [53] (d=2.17) and thus led to a substantial improvement of agoraphobic symptoms, as measured by the PAS.

Table 2 shows the mean values and the standard deviations for the PAS, homework compliance, and positive treatment expectations at different measuring times. It shows both a significant reduction in symptomatology, measured by the PAS, when comparing the baseline data to the date of early change (session 5), t(22)=-5.18, p=0.00, as well as when comparing the time of early change to the end of treatment, t(22)=-5.80, p=0.00.

	Baseline		Point of initial change		Post-CBT
Measure	M	SD	M	SD	M	SD
PAS	29.36	8.96	22	8.51	11.15	7.78
PATHEV	3.26	0.90
Homework compliance			5.41	1.37	5.68	2.14

Note: M=Mean; SD=Standard Deviation; CBT=Cognitive Behavioral Therapy; PAS=Panic and Agoraphobia Scale; PATHEV=Patient Questionnaire on Therapy Expectations and Evaluation.

Table 2: Means and standard deviations for all measures by time point.

Figure 1 shows the development of symptoms, as measured by the PAS, across different sessions. On the one hand, it includes the course of treatment for patients with an early response in sessions 1 to 5 (N=11). On the other hand, it shows the course of treatment for those experiencing improvement after the fifth session (N=12).

Figure 1: Panic and agoraphobia symptoms over the course of treatment. PAS=Panic and Agoraphobia Scale.

Treatment expectancy, homework compliance, and early response

In order to detect a possible multi-collinearity problem, the variance inflation factor (VIF) was determined. The testing showed no multi-collinearity problem. Table 3 summarizes the results of the regression analysis. Our findings did not confirm those of Westra et al. [1], according to which positive treatment expectations were no longer significantly correlated with early response when homework compliance was included in the model. The test of linear regression shows that patients with high positive treatment expectations also showed a higher compliance with homework (a=0.410; KIU=0.215, KIO=0.819), whereas no significant correlation in the linear regression between positive treatment expectations and an early response was found (c=- 1.586; KIU=-5.348, KIO=1.827). In Model 3, in which homework compliance was added as a mediator, this relationship remained not significant (c‘=-1.670; KIU=-6.221, KIO=2.233). A comparison between Models 2 and 3 shows that no additional variance in the dependent variable can be explained by adding homework compliance as a mediator (F(1,19)=0.01, p=0.94). The 95% bootstrap confidence interval for the indirect effect (ab=0.084) included the zero (-1.180, 1.488), thus indicating that the indirect effect did not become significant.

Predictor						95% - CI			Bootstrap 95% - CI
	R2	F	β	B	t	CIL	CIO		CIL		CIO
Model 1: Dependent variable: homework compliance
Positive expectation	0.29	(1,20) = 8.16	0.54	0.41	2.86	0.11		0.71	0.22		0.82
Model 2: Dependent variable: early response
Positive expectation	0.04	(1,20) = 0.89	-0.21	-1.59	-0.94	-5.10	1.93		-5.35	1.83
Model 3:  Dependent variable: early response
Positive expectation Homework compliance	0.04	(2,19) = 0.42	-0.22	-1.67	-0.81	-5.96	2.62		-6.22	2.23
			0.02	0.20	0.08	-5.43	5.84		-4.59	4.73

Note: Homework compliance=mean from sessions 1-5; early response=PAS difference value from sessions T1-S5; R²=explained variance; stand. β=standardized regression weight beta; B=unstandardized regression weight beta; t=proof of the statistical analyses with a t-distribution; 95%-CI=95% confidence interval; bootstrap 95% CI=bootstrapping confidence interval; CIL=lower limit of the 95% confidence interval; CIU=upper limit of the 95% confidence interval.

Table 3: Determination coefficients, standardized and unstandardized regression weighting, and model test for the first mediation hypothesis.

Homework compliance, early response, and treatment outcome

For the following analysis, multi-collinearity was again tested and ruled out. The findings related to the second hypothesis of Westra et al. [1] were also examined and could not be confirmed and were rejected for this reason. The test of linear regression (Table 4) shows, on the one hand, that patients with higher homework compliance did not tend to experience an early change (a=-0.650; KIU=- 4.910, KIO=3.065). On the other hand, it revealed no significant correlation between homework compliance and symptomatology at the end of treatment (c=0.302; KIU=-4.861, n KIO=6.810). Model 2, in which early response was added as a mediator to the model, shows that the mediator was significantly correlated with the dependent variable (b=0.682; KIU=0.058, KIO=1.141). The influence of homework compliance on the overall outcome of the treatment was still not significant (c‘=0.746; KIU=-4.442, KIO=7.473). A comparison between Models 2 and 3 shows that, in the mediation analysis, the additional mediator variable “early response” explained significantly more variance in the dependent variable F(1,20)=5.69, p=0.03). However, the indirect effect (ab=- 0.443) was still not significant as the 95% bootstrap confidence interval included the zero (-3.640, 2.525).

Predictor	R2	F	β	B	t	95% CI		Bootstrap 95% CI
						CIL	CIU	CIL	CIU
Model 1: Dependent variable: early response
Homework compliance	0.00	(1,21) = 0.09	-0.07	-0.65	-0.31	-5.04	3.74	-4.49	3.31
Model 2: Dependent variable: symptoms posttreatment
Homework compliance	0.00	(1,21) = 0.01	0.02	0.30	0.10	-6.06	6.66	-4.74	6.73
Model 3: Dependent variable: symptoms posttreatment
Homework compliance Early response	0.22	(2,20) = 2.85	0.05	0.75	0.27	-5.04	6.53	-4.29	6.79
			0.47	0.68	2.39	0.09	1.28	0.12	1.17

Note: Homework compliance=mean from sessions 1-5; early response=PAS difference value from sessions T1-S5; symptoms posttreatment=PAS difference value from sessions T1-S12; R²=explained variance; stand. β=standardized regression weight beta; B=unstandardized regression weight beta; t=proof of the statistical analyses with a t-distribution; 95% - CI=95% confidence interval; bootstrap 95% CI=bootstrapping confidence interval; CIL = lower limit of the 95% confidence interval; CIU=upper limit of the 95% confidence interval.

Table 4: Determination coefficients, standardized and unstandardized regression weighting, and model test for the second mediation hypothesis.

Discussion

The aim of the present study was to determine the role of pretreatment expectations or compliance with homework influenced treatment outcome. Therefore we investigated whether the findings of Westra et al. [1] could be generalized to other therapy settings [13]. In contrast to in the group intervention of Westra et al. [1], patients with a panic disorder and agoraphobia were treated in an outpatient individual cognitive behavioral therapy with a disorder-specific intervention [36] which included exposure as a central part of the therapy.

Patients who experienced an early change according to the definition in this study (N=19) showed this change on average in session 5.11 (SD=2.94). This finding is consistent with those of other studies, which recorded an early change in the fifth treatment session (e.g., [54-56]).

Our results did not confirm the mediation hypothesis that positive treatment expectancy is no longer significantly correlated with an early change when homework compliance is added to the model. Moreover, our testing also did not confirm the second assumption that homework compliance no longer has a significant relationship with the treatment outcome if early change is included in the model. The bootstrap confidence interval of the indirect effect was not significant for both analyses. However, it was confirmed that early response in the course of therapy has a significant influence on the outcome of therapy. This is in line with previous research findings [23-27].

In the following, various considerations and approaches will be discussed and possible explanations for the deviating findings will be put forward.

First, methodological reasons could be the cause of the deviating findings. In this context, the operationalization of homework compliance plays an essential role. Although homework is an important part of cognitive behavioral therapy [57-58,21], metaanalyses clearly show that findings on the effect of homework depend strongly on how homework compliance is defined and measured [57,21-22]. As this study implemented an improved methodology, using the mean score of homework compliance from the first five sessions, the results are based on a more representative estimator [32].

In addition, the actual content of the study could also have had an influence on the results. For example, the setting is different. People in the study of Westra et al. [1] were treated in a group therapy, whereas patients in the present study were treated with single therapy. There are currently no studies on whether homework in group therapy plays a different role than homework in a single setting. However, it is conceivable that homework is of greater importance for progress in the therapy of an individual, due to peculiarities of grouping such as mutual motivation and reinforcement, model effects by other participants, or less space to pursue an individual issue. At the same time, however, patients in a group setting are not monitored as closely. For example, the therapist is less likely to notice when patients have not completed their homework, whereas in individual therapy this is more likely to be addressed. In addition, the present study clearly shows that interoceptive and in situ confrontational exercises play a special role, since a marked reduction in the symptoms takes place at the beginning of the exercises. For this reason, one might surmise that willingness to face the feared bodily symptoms and the anxiety situation together with the therapist has a more decisive influence on the treatment outcome than homework compliance. In this context, a study by Cammin-Nowak et al. [59] showed that specific homework has a specific effect on the treatment outcome. In their study, in which patients with panic disorder and agoraphobia were also treated according to the manual of Lang et al. [36], the researchers found that exposure-based homework was a better predictor of treatment outcome than homework related to psychoeducation or the development of an individual behavioral analysis. Thus, a high degree of compliance with homework in treatment sessions 6 – 11 appeared to be of greater importance with regard to the therapy outcome. In the present study, in accordance with Westra et al. [1], homework compliance was operationalized based on the average value of sessions 1 -5. In these sessions, patients were initially given psychoeducational contents (sessions 1 – 3) and interoceptive exposure was implemented together with the therapist (sessions 4 – 5). Consequently, it is conceivable that testing the mediation hypotheses might have led to other results. In other words, a correlation between homework compliance and early response might have been confirmed if the mean value of homework compliance computed over all sessions had been included in the analyses (as recommended by Kazantzis et al., [32]. Furthermore, the correlation between homework compliance and the posttreatment might have been confirmed if the mean value for homework compliance from sessions 6 – 11, which integrated in situ exposure, had been used to operationalize homework compliance. Therefore, future studies should also focus on which therapy sessions play a particularly important role in symptom reduction with regard to the operationalization of homework compliance instead of assuming per se that the homework of the first treatment sessions has the greatest impact on the outcome of therapy.

Furthermore, there are several weaknesses of the findings of Westra et al. [1]. First, it is not clear as to how effective the treatment is. The researchers merely refer to an unpublished manuscript and do not report on the effect of the therapy. In their study, group therapy was implemented to treat patients with panic disorder and agoraphobia as well as socially phobic patients with a generalized anxiety disorder. It can therefore be assumed that it was not a disorder-specific treatment manual. It is also questionable which specific homework was part of the therapy or especially after the second treatment session. In contrast, patients who participated in the present study were treated with a specific therapeutic manual for panic disorder with agoraphobia [36]. This treatment manual has been found to be highly effective [7], also showing a high effect in this study (d=2.17) and a reduction of the symptoms comparable to previous studies [7].

Second, the results presented by Westra et al. [1] should be assessed critically, since testing the first hypothesis may have permuted the statistical characteristics (Table 3). The representation of the reported R2 and F-values from step 2 of the analyses appears to be flawed, as revealed by a direct comparison to step 3. Therefore it is unclear as to whether the reported results that homework compliance fully mediates the relationship between positive treatment expectations and an early response actually exist, or as to whether no significant indirect effect can be found, as in the present findings.

Nevertheless, the results of this study must also be interpreted carefully as it is subject to some limitations. First, although the sample characteristics are representative for panic disorder with agoraphobia, the small sample size limits the conclusion drawn.

Second, the study did not have a waiting list control group, which is usually used to investigate whether symptom reduction is the result of the specific intervention or whether there would have been a natural improvement over time. Based on data on the chronification of the disease symptomatology, this assumption is rather unlikely.

Third, patients rate their homework compliance as very high throughout the therapy. Homework compliance at the time of early response was M=5.41, SD=1.37 out of a maximum value of 7 and at the end of therapy it was M=5.68, SD=2.14. Therefore, variance in this variable and in the predictor variable “positive treatment expectations” (M=3.26, SD=0.90 of a maximum value of 4) is restricted.

And finally, like in the study of Westra et al. [1], we only examined the influence of positive treatment expectations on homework compliance. Other studies show that higher positive expectations on the therapy result in greater involvement and commitment of the patient and, thus, in a better treatment outcome [11,17-18,1]. Indeed, many other variables could have an impact on homework compliance, for example the working relationship between the patient and the therapist. Studies on the therapeutic alliance not only report impressive efficacy of up to 0.64, but also confirm that the therapeutic relationship is more frequently associated with the success of psychotherapy than monistic variables, that is characteristics only attributed to the therapist or to the patient as an individual [60,61].

Despite these limitations, the results of the present study are an extension of current research findings and provide an initial indication that the homework compliance of a patient with panic disorder and agoraphobia undergoing cognitive behavioral therapy with exposure-based interventions is not a primary predictor variable for the treatment outcome. Indeed, our results suggest that interoceptive and in situ exercises lead to a clear reduction in symptoms.

As a conclusion of the described results, it is important in the future that studies of patient characteristics that could be responsible for the success or failure of a treatment contribute to the understanding of processes in psychotherapy and make a valuable contribution toward improving the effectiveness of psychotherapeutic interventions. Therefore, more research is required, especially to replicate and confirm previous findings, and expand the scientific state of knowledge. The repetition of a study shows that findings, such as those described by Westra et al., [1] albeit of great importance, cannot be applied to all settings. In the literature, risk factors have been identified that impair replicability such as small samples, poor definition of the outcome and data analysis, low treatment integrity, ignoring therapists effects and overestimating small effects. This factor may also play a decisive role in the primary study of Westra et al. [1] and the present study [28]. For this reason, there should be an increased interest in replication studies in order to develop psychological theories and generalize research findings [13].

References

1. Westra HA, Dozois DJA, Marcus M (2007) Expectancy, homework compliance, and initial change in cognitive-behavioral therapy for anxiety. J Consult Clin Psychol 75: 363-373.
2. Ruhmland M, Margraf J (2001) Effectiveness of psychological therapies of panic and agoraphobia: meta-analyzes on disturbance level. Behav Ther 11: 41–53.
3. Sánchez-Meca J, Rosa-Alcázar AI, Marín-Martínez F, Gómez-Conesa A (2010) Psychological treatment of panic disorder with or without agoraphobia: a meta-analysis. Clin Psychol Rev 30: 37–50.
4. Stewart RE, Chambless DL (2009) Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: a meta-analysis of effectiveness studies. J Consult Clin Psychol 77: 595–606.
5. Barlow DH (1990) Long-term outcome for patients with panic disorder treated with cognitive-behavioral therapy. J Clin Psych 51: 17–23.
6. Öst LG, Thulin U, Ramnerö J (2004) Cognitive behavior therapy vs exposure in vivo in the treatment of panic disorder with agrophobia. Behav Res Ther 42: 1105–1127.
7. Gloster AT, Wittchen HU, Einsle F, Lang T (2011) Psychological treatment for panic disorder with agoraphobia: a randomized controlled trial to examine the role of therapist-guided exposure in situ in CBT. J Consult Clin Psychol 79: 406–420.
8. Cohen M, Beard C, Björgvinsson T (2015) Examining patient characteristics as predictors of patient beliefs about treatment credibility and expectancies for treatment outcome. J Psychother Integr 25: 1–10.
9. Greenberg RP, Constantino MJ, Bruce N (2006) Are patient expectations still relevant for psychotherapy process and outcome? Clin Psychol Rev 26: 657–678.
10. Price M, Anderson PL (2012) Outcome expectancy as a predictor of treatment response in cognitive behavioral therapy for public speaking fears within social anxiety disorder. Psychotherapy 49: 173–179.
11. Arnkoff DB, Glass CR, Shapiro SJ (2002) Expectations and preferences. In JC Norcross, editors. Psychotherapy relationships that work: therapist contributions and responsiveness to patients. New York: Oxford University Press. pp: 325–346.
12. Open-Science-Collaboration (2015) Estimating the reproducibility of psychological science. Science 349: aac4716.
13. Pashler H, Harris CR (2016) Is the replicability crisis overblown ? three arguments examined. Perspect Psychol Sci 7: 531–536.
14. Constantino MJ, Arnkoff DB, Glass CR, Ametrano RM, Smith JZ (2011) Expectations. J Clin Psychol 67: 184–192.
15. Bandura A (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 84: 191–215.
16. Safren SA, Heimberg RG, Juster HR (1997) Clients’ expectancies and their relationship to pretreatment symptomatology and outcome of cognitive-behavioral group treatment for social phobia. J Consult Clin Psychol 65: 694–698.
17. Chambless DL, Tran GQ, Glass CR (1997) Predictors of response to cognitive-behavioral group therapy for social phobia. J Anxiety Disord 11: 221–240.
18. Ilardi SS, Craighead WE (1994) The role of nonspecific factors in cognitive-behavior therapy for depression. Clin Psychol 1: 138–155.
19. Chu BC, Kendall PC (2004) Positive association of child involvement and treatment outcome within a manual-based cognitive-behavioral treatment for children with anxiety. J Consult Clin Psychol 72: 821–829.
20. Chu BC, Kendall PC (2009) Therapist responsiveness to child engagement: flexibility within manual-based CBT for anxious youth. J Clin Psychol 65: 736–754.
21. Kazantzis N, Whittington C, Dattilio F (2010) Meta-analysis of homework effects in cognitive and behavioral therapy: a replication and extension. Clin Psychol 17: 144–156.
22. Mausbach BT, Moore R, Roesch S, Cardenas V, Patterson TL (2010) The relationship between homework compliance and therapy outcomes: an updated meta-analysis. Cognit Ther Res 34: 429–438.
23. Penava SJ, Otto MW, Maki KM, Pollack MH (1998) Rate of improvement during cognitive-behavioral group treatment for panic disorder. Behav Res Ther 36: 665–673.
24. Clerkin EM, Teachman BA, Smith-Janik SB (2008) Sudden gains in group cognitive-behavioral therapy for panic disorder. Behav Res Ther 46: 1244–1250.
25. Lutz W, Hofmann SG, Rubel J, Boswell JF (2014) Patterns of early change and their relationship to outcome and early treatment termination in patients with panic disorder. J Consult Clin Psychol 82: 287–297.
26. Crits-Christoph P, Connolly MB, Gallop R, Barber JP, Tu X, et al. (2001) Early improvement during manual-guided cognitive and dynamic psychotherapies predicts 16-week remission status. J Psychother Pract Res 10: 145–154.
27. Schlagert H, Hiller W (2015) Frequency and predictive value of early response in naturalistic psychotherapies. Z Klin Psychol Psychother (God) 44: 159–168.
28. Leichsenring F, Steinert C (2016) Replicability in psychotherapy research: uncontrolled risk factors and proposed measures. Psychother Psychosom 85: 371–372.
29. Glenn D, Golinelli D, Rose RD, Roy-Byrne P (2013) Who gets the most out of cognitive behavioral therapy for anxiety disorders? the role of treatment dose and patient engagement. J Consult Clin Psychol 81: 639–649.
30. Addis ME, Jacobson NS (2000) A closer look at the treatment rationale and homework compliance in cognitive-behavioral therapy for depression. Cogn Ther Res 24: 313–326.
31. Leung AW, Heimberg RG (1996) Homework compliance, perceptions of control, and outcome of cognitive-behavioral treatment of social phobia. Behav Res Ther 34: 423–432.
32. Kazantzis N, Deane FP, Ronan KR (2004) Assessing compliance with homework assignments: review and recommendations for clinical practice. J Clin Psychol 60: 627– 641.
33. Bandelow B (1997) Panic and agoraphobia scale (PAS). Seattle, WA: Hogrefe & Huber.
34. Peterson RA, Reiss S (1992) Anxiety sensitivity index manual. 2nd Edn Worthington, OH: International Diagnostic Systems.
35. Wittchen HU, Wunderlich U, Gruschwitz S, Zaudig M (1997) Structured clinical interview for DSM-IV. Axis I: Mental Disorders. University of Göttingen: Hogrefe Verlag.
36. Lang T, Helbig-Lang S, Westphal D, Gloster AT, Wittchen HU (2011) Exposure-based therapy of panic disorder with agoraphobia: a treatment manual. Goettingen: Hogrefe.
37. American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders. 4th Edn. Washington, DC: American Psychiatric Association.
38. Kenardy J, Dow MGT, Johnston DW, Newman, MG, Thomson A, et al. (2003) A comparison of delivery methods of cognitive-behavioral therapy for panic disorder: an international multicenter trial. J Consul Clin Psychol 71: 1068-1075.
39. Hoyer J, Helbig S, Margraf J (2005) Diagnosis of anxiety disorders. Compendia psychological diagnostics. Göttingen: Hogrefe.
40. Bandelow B, Brunner E, Broocks A, Beinroth D (1998) The use of the panic and agoraphobia scale in a clinical trial. Psychiatry Res 77: 43–49.
41. Schulte D (2005) Measurement of therapy expectation and therapy evaluation of patients (PATHEV). J Clin Psychol Psychother 34: 176–187.
42. Schulte D (1993) How should success be measured? overview work. Z Klin Psychol Psychother (God) 22: 374-393.
43. Cohen J, Cohen P, West SG, Aiken LS (2003) Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, New Jersey, London: Lawrence Earlbaum Associates.
44. Gollwitzer M, Christ O, Lemmer G (2014) Individual differences make a difference: On the use and the psychometric properties of difference scores in social psychology. Eur J Soc Psychol 44: 673–682.
45. https://www.r-project.org
46. Baron RM, Kenny DA (1986) The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol 51: 1173–1182.
47. MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V (2002) A comparison of methods to test mediation and other intervening variable effects. Psychol Methods 7: 83–104.
48. Shrout PE, Bolger N (2002) Mediation in experimental and nonexperimental studies: new procedures and recommendations. Psychol Methods 7: 422–445.
49. Collins LM, Graham JW, Flaherty BP (1998) An alternative framework for defining mediation. Multivar Behav Res 33: 295–312.
50. Mackinnon DP, Lockwood CM, Williams J (2004) Confidence limits for the indirect effect: distribution of the product and resampling methods. Multivar Behav Res 39: 99.
51. Eid M, Gollwitzer M, Schmitt M (2010a) Statistics and research methods. Wine: Beltz pp: 926–954
52. Eid M, Gollwitzer M, Schmitt M, editors (2010b) Statistics and research methods. Wine: Beltz pp: 191–249
53. Cohen J (1988) Statistical power analysis for the behavioral sciences. 2nd Edn. Hillsdale, NJ: Lawrence Earlbaum Associates.
54. Hofmann SG, Schulz SAM, Meuret AE, Moscovitch DA, Suvak M (2006) Sudden gains during therapy of social phobia. J Consult Clin Psychol 74: 687–697.
55. Kelly Ma R, Roberts JE, Ciesla JA (2005) Sudden gains in cognitive behavioral treatment for depression: when do they occur and do they matter? Behav Res Ther 43: 703–714.
56. Tang TZ, DeRubeis RJ (1999) Sudden gains and critical sessions in cognitive - behavioral therapy for depression. J Consult Clin Psychol 67: 894–904.
57. Kazantzis N, Deane FP, Ronan KR (2000) Homework assignments in cognitive and behavioral therapy: a meta-analysis. Clin Psychol 7: 189–202.
58. Kazantzis N, Lampropoulos GK, Deane FP (2005) A national survey of practicing psychologists’ use and attitudes toward homework in psychotherapy. J Consult Clin Psychol 73: 742–748.
59. Cammin-Nowak S, Helbig-Lang S, Lang T, Gloster AT, et al. (2013) Specificity of homework compliance effects on treatment outcome in CBT: evidence from a controlled trial on panic disorder and agoraphobia. J Clin Psychol 69: 616-629.
60. Horvath AO, Symonds BD (1991) Relation between working alliance and outcome in psychotherapy: a meta-analysis. J Couns Psychol 38: 139–149.
61. Martin DJ, Garske JP, Davis MK (2000) Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol 68: 438–450.