Introduction

Encouraging clients’ use of therapy skills and engagement with homework assignments between therapy sessions is an integral component of cognitive behavior therapy (CBT) and other manualized treatments (Detweiler and Whisman 1999; Gaynor et al. 2006; Kazantzis et al. 2000; Rees et al. 2005). Many theorize that in order to succeed in therapy, clients must practice the skills they learn in the therapy session (Coon and Thompson 2003; Detweiler and Whisman 1999). According to Beck et al. (1985), homework allows for clients to experiment and learn from their thoughts, behaviors, and feelings in real-life situations. In CBT, these everyday personal experiences are crucial to changing belief structures as they lead to increased reinforcing events that challenge negative thought patterns (Addis and Jacobson 2000; Detweiler and Whisman 1999). Kazantzis and Deane (1999), reported that 98% of therapists surveyed incorporated homework into their treatment. Homework assignments are often used by therapists to encourage the use of therapy skills in between therapy sessions, although it should be noted that homework compliance and skill use are not the same thing. For example, a client can be “compliant” with homework by filling out worksheets in a cursory manner the night before a session without ever using the skills from therapy in daily life.

The completion of homework assigned in therapy also increases the clients’ ability to generalize therapy skills throughout their lives and increases their sense of self-efficacy (Burns and Spangler 2000; Gaynor et al. 2006; Detweiler and Whisman 1999). Additionally, homework compliance demonstrates client openness, engagement in therapy, and ownership of their improvement (Addis and Jacobson 2000; Beck et al. 1985; Detweiler and Whisman 1999). Unfortunately, the impact and possible benefits of homework compliance in therapeutic outcomes are often not prioritized in research studies (Burns and Spangler 2000; Detweiler and Whisman 1999; Rees et al. 2005).

A meta-analysis of 27 studies indicated that therapy that utilized homework lead to better treatment outcomes than therapy without homework assignments (Kazantzis et al. 2000). Further, homework compliance has been found to be a significant predictor of early and post-treatment outcomes, such as decreased depression and anxiety scores (Addis and Jacobson 2000; Coons and; Thompson 2003; Kazantzis et al. 2000; Rees et al. 2005). Specifically, engaging in behavioral tasks was found to lower depressive symptoms and increase quality of life, and completion of CBT thought records significantly predicted reduced anxiety scores (Rees et al. 2005). A study using structural equation modeling demonstrated the causal impact of homework on changes in depression, controlling for demographics, therapist identities, therapeutic empathy, motivation, and other variables influencing outcome over and above homework compliance (Burns and Spangler 2000; Kazantzis et al. 2001).

Unfortunately, methods for measuring homework completion vary widely (Gaynor et al. 2006; Kazantzis et al. 2004), and as noted previously, homework compliance is not necessarily synonymous with therapy skill use. The majority of research studies have relied on therapist or client reports of homework completion after therapy was completed, instead of session by session measurement (Detweiler and Whisman 1999; Kazantzis et al. 2004). For example, Burns and Spangler’s (2000) study on causality relied on retrospective therapist and patient report of compliance at 12 weeks. A small handful of studies have measured homework compliance more frequently, for example through the use of therapist rating at each session (Addis and Jacobson 2000; Coon and Thompson 2003). Studies involving daily records from clients may provide an even more accurate assessment of skill use. For example one study used diary card reports in dialectical behavior therapy (e.g., Lindenboim et al. 2007) and another used a measure in which participants provided daily records of homework completion (a “yes/no” client report of skill use; Rees et al. 2005). While these more frequent assessments are an improvement over measures that may be more prone to retrospective biases, paper-and-pencil approaches are typically not time- and date-stamped, so the possibility of retrospective reporting cannot be ruled out.

Methodological Considerations

Previous studies seem to clearly suggest that homework completion is important in therapy, and there is an assumption that clients who are more adherent with homework are using the skills they are learning in therapy to help them cope with problems in daily life. However, we know very little about what clients actually do—what skills they use—in their daily lives in order to cope with their difficulties. Most of the studies examining homework completion have not been designed to look at specific skill use in daily life. Kazantzis et al. (2000), reported that four studies discussed relaxation, three exposure, two assertiveness skills, one biofeedback, and one video homework. Only a few studies, to our knowledge, described the type of homework assignments in their methods. For example, some studies focused on thought records, pleasurable activities, and behavioral activation (Busch et al. 2010; Coon and Thompson 2003; Rees et al. 2005). However, none of these studies were designed to measure skill use in participants’ daily life.

Experience sampling and diary methods, in which data are collected frequently over the course of therapy in the client’s natural environment, are well-suited for examining skill use in daily life. When data are collected using tablets, phones, or computers, responses are time- and date-stamped, allowing the researcher to verify the timing of responses (Bolger and Laurenceau 2013; Conner et al. 2009; Hektner et al. 2007). Surprisingly, despite their obvious advantages, experience sampling and diary methods have rarely been used to look at skill use among clients undergoing psychotherapy. The primary goal of the current study was to use quasi-daily diary methods to look at the impact of the use of skills learned in therapy on daily functioning and mood using data from a small randomized controlled trial comparing two forms of psychotherapy for major depression: CBT and self-system therapy (SST). Although these two treatments are similar in some ways, for example both are short-term psychotherapies focused on developing skills through in-session discovery and between session homework, the underlying theories are quite different. Whereas CBT focuses on altering distorted beliefs, SST focuses on self-regulatory skills, for example by teaching people how to more effectively set and pursue goals and evaluate goal progress. Both CBT and SST have been shown to be effective treatments for depression (Eddington et al. 2015; Strauman et al. 2006). We chose to monitor skill use rather than homework compliance because, while a specific homework assignment in a given week may not necessarily involve daily work, both therapies emphasize building a set of skills (cognitive or self-regulatory) over the course of therapy and integrating those skills into daily life.

Participants in this study were instructed to complete a phone-based survey two times per week throughout the course of short-term therapy using an interactive voice response (IVR) system. Each survey included questions about the use of therapy skills that day as well as ratings of mood, activities, and other aspects of daily functioning. In addition, each survey included an open-ended question in which participants were asked to describe the skills they had actually used that day. We opted for an open-ended format in order to obtain participants’ own descriptions of what they did, rather than relying on checklists, and coded responses into five categories (cognitive skills, self-regulatory skills, relaxation, other, or no skill use).

In this study, we hypothesized that, on the days participants report higher levels of skill use, they would report higher emotional well-being and better overall functioning. Additionally, we hypothesized that participants in the CBT condition would report using more cognitive skills, whereas participants in the SST condition would report the use of more self-regulation skills. Therefore, given that previous studies have focused on the overall benefit of therapy skill use and homework completion on therapeutic outcome, our study aims to expand on existing literature by demonstrating the impact of specific daily skill use on well-being.

Methods

Participants

For a detailed description of recruitment procedures, inclusion criteria, treatment condition assignment, and participant flow, see Eddington et al. (2015). Of the 56 adults with primary major depressive disorder or dysthymia who did not meet study exclusionary criteria (which included current use of antidepressant medications) and were assigned to either SST (n = 27) or CBT (n = 29), six failed to return for any therapy sessions and four did not respond to any of the diary surveys. The final sample, therefore, included 46 participants randomized to SST (n = 22) or CBT (n = 24). Furthermore, to be included in current analyses, participants had to have completed greater than six intermittent calls, have more than four therapy sessions completed, and have <50% of calls relative to number of sessions completed. Thirty-nine people met these criteria and thus were included in the intermittent call analysis (22 in CBT and 17 in SST). Sample descriptives are shown in Table 1. Of note, two participants of the thirty-nine did not provide any verbal responses to the open-ended question, and were therefore not included in the skills coding analyses (final sample of 21 in CBT and 16 in SST).

Table 1 Descriptive information of participants
Full size table

Procedures

Participant eligibility was determined during an initial laboratory assessment during which participants completed the Beck depression inventory-II (BDI-II) and Beck anxiety inventory (BAI). If the BDI-II score was above a 14, participants underwent semi-structured diagnostic interviews; participants were eligible for enrollment if they met DSM-IV-TR diagnostic criteria for major depressive disorder or dysthymia (see Eddington et al. 2015, for enrollment details) and did not have any exclusionary conditions (no current antidepressant medications, no history of mania, no current substance dependence, no history of psychotic symptoms, no diagnosis of antisocial or borderline personality disorders, and no active suicidal intent). Once enrolled in the study, participants began the intermittent diary surveys during the week of their first therapy session. A phone-based survey system (Telesage 2009) called each participant on 2 random days per week for the entire course of therapy with a reminder to complete the diary survey any time on that day. Responses were all time- and date-stamped in order to confirm completion with the sampling procedure.

Details about the two therapy conditions (including therapist training and adherence outcomes) can be found in Eddington et al. (2015). Briefly, both treatments involved 16 sessions of therapy, and both treatments were structured and focused on teaching skills for coping with current life situations through in-session work as well as homework assignments. In CBT, the focus is on cognitive restructuring skills and modifying core beliefs; in SST, the focus is on skills supporting effective self-regulation. Note that, due to scheduling issues, therapy sessions did not always occur weekly (although one session per week was the aim). Therefore, the number of weeks of data collection often exceeded 16.

Measures

Beck Depression Inventory-II

The BDI-II (Beck et al. 1996) is a widely used self-report measure of depressive symptom severity during the past 2 weeks. The scale contains 21 items and each is rated on a 4-point scale (0–3), with total scores ranging from 0 to 63. The BDI-II has excellent psychometric properties (Dozois and Covin 2004), and the internal consistency in this study was good (Cronbach’s alpha = 0.87).

Beck Anxiety Inventory

The BAI (Beck et al. 1988) is a widely-used self-report measure of anxiety symptom severity during the past 2 weeks. The scale consists of 21 items and each is rated on a 4-point scale (0–3) scale, with total scores ranging from 0 to 63. The BAI has good psychometric properties (Fydrich et al. 1992); in this study, internal consistency was very good.

Diary Methodology—Items

See Table 2 for the full list of the diary survey items. Each item was rated by the participant using a phone number pad on a scale from one (“not at all”) to seven (“very much”) with a midpoint of four (“somewhat”).

Table 2 Intermittent diary items
Full size table

Skill Use Coding

During the phone survey, participants were asked to verbally respond to the question, “What skills did you use today?” In addition, they were asked to rate how helpful the skills were for them, using the same one to seven scale described above. Audio recorded answers were coded for the types of skills used; coders were unaware of the assigned treatment condition for the participants.Footnote 1 Responses were coded (1 = skill category used or 0 = skill category not used) by a trained coder, AH, based on five possible categories: cognitive, self-regulation, relaxation/mindfulness, other, or did not use any skills. A written description of each category was developed and was used as a reference throughout the coding process. The cognitive category included mentions of rethinking, beliefs, or thinking positively. For example, “I wrote down my thought process” or “I came up with alternative/realistic thoughts.” Self-regulation included doing things for pleasure, focusing on goals, making plans, or staying busy, such as, “I accomplished even small things on my list,” “I focused on promotion goals,” or “I created an action plan.” The relaxation/mindfulness category included mentions of relaxing, deep breathing, progressive muscle relaxation, or living in the present moment. These included statements such as, “I relaxed after work,” or “I practiced deep breathing.” Skills that did not fit into these categories were coded as “other.” Occasionally, participants reported using multiple skills in a single response that fell into different categories (15 calls out of 634, 2.4%). In those cases, all relevant categories were coded for that response; the maximum number of categories used in a single response was two.

In order to assess the reliability of the coding, a random selection of 20% of the first 220 audio files was coded by a second coder (KE). There was only one minor disagreement concerning a response that involved two categories; the coders agreed on one of the categories but differed on the other, and a consensus was quickly reached. Because the reliability was excellent (98%), the remainder were coded by a single coder.

Data Analytic Approach

Ratings made in daily life (within-person level) are nested within participants (between-person level) therefore, multilevel modeling is an appropriate statistical technique for analyzing this type of data (Affleck et al. 1999). Multilevel modeling is an extension of the more commonly used multiple regression analyses (Hox 2002), and is standard practice for the analysis of repeated, longitudinal data. Our current analyses focused on examining the relationship between participants’ responses to the question “Today I tried using some skills I’ve learned in therapy” with various other measures. As such, the coefficients produced by these analyses are comparable to unstandardized regression coefficients. For the current study, all analyses were conducted using Mplus 7 (Muthén and Muthén, 1998–2010). In addition, within-person predictors were centered at each person’s own mean (Luke 2004) and we calculated parameter estimates using maximum likelihood estimation with robust standard errors. Finally, the analyses were run as a random effects model.Footnote 2

Aggregate Construction

Given participant time constraints, constructs measured in daily life are typically assessed using only one or two questions (Fisher and To 2012). However, past research has shown that the mean of several daily life items can be calculated to form an aggregate score to be used as a more robust assessment of the construct (e.g., Burgin et al. 2013). Consistent with past research, the current study calculated three aggregate mean scores to better measure daily life experiences. The first, positive affect aggregate was computed by calculating the mean of responses to the following items: I felt happy today, I felt enthusiastic today, I felt in control of my emotions today, I felt satisfied today, I felt relieved today, I felt proud today, I felt confident today. The second, negative affect aggregate, was calculated by taking the mean of responses to the following items: I felt sad today, I felt guilty today, I felt anxious today, I felt like a failure today, I felt irritable today, I felt discouraged today. Finally, negative somatic symptoms was computed by taking the mean of the responses to the following items: I felt tired today, I didn’t feel well today, I had a lot of energy today (reverse coded), I had a lot of body aches and pains today, my sleep was restful last night (reverse coded), I slept more today than I should have.Footnote 3

Results

Daily Life Results

Results from Table 3 demonstrate that responses to the daily life question “Today I tried using some skills I’ve learned in therapy” (hitherto referred to as the skills question) related positively with a positive affect aggregate that was created by calculating the mean of the responses to seven other daily life questions that tapped into positive affect. Conversely, responses to this question were negatively associated with a negative affect aggregate that consists of six other daily life questions which tapped into negative affect. With regard to physical functioning, responses to the skills question were negatively associated with a six item aggregate measuring negative somatic symptoms.

Table 3 Relationship of responses to the question “Today I tried using some skills I’ve learned in therapy” with other daily life criterion
Full size table

Next, we examined social functioning (see results in Table 3). Scores on the skills question positively predicted enjoyment of being around others, interacting with friends and family in person and by other means (e.g., phone, text, online), but was unrelated to how concerned participants were about how others thought of them that day. Responses to the skills question were also positively related to questions pertaining to positive activities. Specifically, responses to the skills question positively predicted follow through with daily obligations and feelings of making progress towards goals. Conversely, it was negatively associated with feelings of having to push to get anything done and avoiding things that needed to be done. After giving an open-ended response to the question “The best thing I did today was”, participants then rated how pleasant or enjoyable the activity was. The skills question scores positively predicted responses to this question. However, responses to the skills question did not predict responses to follow up question “How unpleasant was it” which was a follow up question to the open-ended question “The worst thing I did today was”. Furthermore, responses to the skills question also did not significantly predict responses to the question, “How stressful was it” which was a follow up question to the open-ended question “The most stressful thing that happened to me today was”.

Next, we examined the relationship between responses to the skills question with questions that measured cognitive functioning in daily life (see Table 3). Results demonstrate that skills question scores were negatively associated with having trouble concentrating in daily life and having negative or upsetting thoughts that wouldn’t go away. Conversely, responses to the skills question positively predicted thinking about daily life accomplishments. However, scores to the skills question were unrelated to participants stating that they compared themselves to other people, did not worry about things, and keeping the mind focused. Finally, responses to the skills question positively predicted therapy-related questions. Specifically, regarding skills learned in therapy, participants’ skills question scores positively predicted feeling that the skills learned during therapy were helpful and the belief that their depression symptoms would improve.

Open-Ended Reports of Skill Use

For the intermittent phone question regarding the report of skills used that day, 634 audio answers were obtained from 37 participants. Audio answers were coded into the cognitive (n = 149), self-regulation (n = 117), relaxation/mindfulness (n = 45), and other (n = 159) skills categories. The other category included skills such as “being with family,” “removed myself from the situation when I was angry,” or “help others.” Additionally, the other category included the mention of “doing the homework assigned during therapy” without providing further detail. Of the 634 audio calls recorded, 180 (28%) described not using skills due to only having one or two therapy sessions, lack of motivation, or not needing skills that day.

Additionally, an independent-samples t test was conducted to compare the categories of skills used by participants in the SST and CBT conditions. Consistent with our predictions, results indicated participants in the CBT condition reported using more skills coded into the cognitive category and participants in the SST condition reported using more skills coded into the self-regulation category (see Table 4). No significant differences between conditions were found for the relaxation/mindfulness, other, and did not use any skills. Across conditions, participants on average reported that they found the use of therapy skills to be moderately helpful (M = 3.92; SD = 1.26).

Table 4 Mean comparisons and descriptives for participants in CBT and SST on categories of skills used
Full size table

Discussions

The use of therapy skills and completion of between-session assignments has been demonstrated to be a key component of positive therapeutic outcomes (Addis and Jacobson 2000; Coon and Thompson 2003; Kazantzis et al. 2000; Kazantzis and Lampropoulos 2002; Rees et al. 2005). Unfortunately, methods of measuring homework compliance have often relied on therapist or client reports after therapy was completed (Burns and Spangler 2000; Detweiler and Whisman 1999; Kazantzis et al. 2004). While some studies have used more frequent assessments of skill use such as diary cards (e.g., Lindenboim et al. 2007), there is typically no assurance that these written self-reports are actually completed on the day and time recorded by the respondent. This study is the first, to our knowledge, to use a phone-based diary method, which provided time- and date-stamped responses, to assess the relation between the use of skills learned in therapy and daily functioning and mood.

Results indicated that, on days that participants reported greater use of skills learned in therapy, they also reported higher positive affect, lower negative affect, better cognitive functioning, and more effective and satisfying activities (e.g., more enjoyment interacting with others, less avoidance of obligations, and greater progress on personal goals). These results underscore findings from previous studies that highlight the benefit of between-session homework completion during therapy. However, while many previous studies have assessed skill use by monitoring or inquiring about homework completion, our diary approach did not ask about homework per se. Rather, participants were asked about skills learned in therapy that they had used on the day of each survey; those skills may or may not have been associated with a homework assignment for that week. Additionally, this study expanded on previous knowledge by demonstrating the connection that can exists between daily skill use and mood and functioning in between sessions. In other words, while previous studies reported on benefits to well-being at the end of therapy, this study can speak to daily changes that occur throughout therapy.

As expected, coding of audio recorded answers demonstrated that participants in the CBT condition tended to use more cognitive skills, whereas participants in the SST condition reported using more self-regulation skills. This finding suggests that participants were able to integrate these skills in their daily life. This differentiation between the two therapy conditions is not surprising, but to our knowledge no previous study has inquired about skill use in an open-ended way, allowing participants to describe in their own words the strategies they had used. An interesting expansion of this finding would be to determine the extent to which those self-reported strategies correspond directly to homework assignments or session-related material. In other words, do clients experiment with new strategies on pace with progress in therapy, or do they “latch on” to one strategy early on and stick with that?

Of note, the current results are correlational. Therefore, conclusions cannot be made about whether participants demonstrated improved mood because of skill use or if those participants who were feeling better were more likely to use the skills or more likely to report the use of skills. The issue of causality in homework completion research has been raised in previous studies. For example, Burns and Spangler (2000) examined direction of causality in two large samples of depressed patients undergoing CBT; they concluded that homework compliance has a causal effect on reduction in depression and not vice versa. An additional limitation of this study is the generalizability of the results due to the small sample size. Further, the potential therapeutic effects of experience sampling and diary methods have been noted (e.g., Telford et al. 2012), and frequent questions about mood may enhance awareness of subtle changes. The lack of a control condition limits our ability to determine the extent to which the observed changes in daily functioning are attributable to therapy itself versus the diary survey process. Finally, this study focused on skill use of participants in either CBT or SST. Given that acquiring new skills is central to both modes of therapy, future studies focusing on other styles of therapy may be beneficial in order to identify different patterns.

In conclusion, the use of diary methods in this study allowed us to add to the current literature by not only understanding what changes are occurring in therapy but how those changes may be occurring. This methodology demonstrated what clients actually do in their daily lives to cope with their difficulties and how the use of specific skills learned in therapy are related to well-being. The growing popularity of methods that allow for context-sensitive assessment suggests that these methods will soon become standard components of treatment outcome studies. Thus, it is important to establish both their feasibility and validity in various diagnostic groups.