Outcome Assessment in Residential Treatment

Steven I Pfeiffer

• Publisher: Routledge
• Year: 2014
• eBook ISBN: 9781317837251
• Topic: Medicine
• Subtopic: Medical Theory, Practice & Reference

As residential treatment centers and psychiatric hospitals are increasingly asked to document their effectiveness, it is essential for mental health care providers to demonstrate the efficacy and cost-effectiveness of the services they provide. Outcome Assessment in Residential Treatment helps health care providers demonstrate that their planned treatment is necessary and active rather than simply custodial. A practitioner's guide to conducting treatment outcome assessment projects, this innovative book presents readers with historical perspectives, current issues, and practical suggestions for implementing an outcome assessment project.Outcome Assessment in Residential Treatment guides psychiatrists, psychologists, mental health practitioners, and social program administrators in addressing which therapeutic components contribute to the goals and objectives of their programs and which may require modification, radical revision, or even elimination. It helps residential treatment centers and psychiatric treatment facilities document treatment successes and better understand which factors (within the client, family, environment, treatment setting, or combinations therein) predict successful outcome. This objective data empowers readers to influence government and industry, enhance public awareness of the needs of severely disturbed children and youth, and validate the usefulness of intensive psychiatric treatment.Unlike other books on treatment outcome, Outcome Assessment in Residential Treatment tells readers how to determine clinically significant improvement and not simply statistically significant change. It gives practical, detailed, proven advice on how to carry out studies that will benefit residential treatment centers and the psychiatric and mental health fields. Contributors provide tools to validate/demonstrate that psychiatric and mental health treatments are effective. They offer insight into:

• planning a treatment outcome project
• recognizing ethical, practical, methodological, logistical, and clinical considerations in implementing a treatment outcome project
• selecting instruments to assess treatment outcome and measuring success
• comparing different outcome measuresHealth care providers must have accurate information about treatment outcomes to demonstrate that specific services are beneficial, cost-effective, and well-received by the client. Outcome Assessment in Residential Treatment helps readers evaluate the impact a treatment program has on a client's clinical status and psychosocial and educational functioning, making it possible to provide an objective yardstick for the payer's evaluation of the quality of care provided.Psychiatrists, psychologists, mental health practitioners, and social program administrators will find Outcome Assessment in Residential Treatment an essential guide to evaluating and understanding the relative effects of specific interventions or procedures on the quality and effectiveness of their services. They will use this information to make appropriate changes which guarantee that they best meet their clients'mental health care needs.

Introduction to Treatment Outcome: Historical Perspectives and Current Issues

Sarah I. Pratt, PhD

Kevin L. Moreland, PhD

SUMMARY. Outcome research on child and adolescent mental health services serves many important purposes, the most important of which is to improve upon the efficacy of techniques that may be used to alleviate the suffering of countless troubled youngsters. The current body of research in this area is small compared to the body of research on the outcomes of adult psychotherapy, perhaps because, historically, the treatment of childhood mental disorders has not received as much attention. Meta-analyses of outcome studies conducted over the past twenty years have concluded that research has demonstrated the efficacy of child and adolescent psychotherapy; however, recent evaluations of these meta-analyses have warned that most studies have included features that may not be generalizable to treatment as actually practiced in clinical settings. Measuring and defining improvement in any population is complicated and there is, as yet, no consensus on standards. For several reasons, conducting research on children and adolescents can be particularly problematic. Although significant strides have been made in child and adolescent mental health care, further research must include standardization of measures of improvement and finer-grained analyses of a greater number of the variables that may influence the efficacy of treatment. [Article copies available from The Haworth Document Delivery Service: 1-800-342-9678. E-mail address: [email protected]]

Progress in the diagnosis, understanding and treatment of child and adolescent psychopathology has advanced considerably in recent years. Many researchers complain that outcome research on child and adolescent mental health services, particularly hospital treatment, lags behind research on adult treatment (Casey & Berman, 1985; Colson, Cornsweet, Murphy, O’Malley, Hyland, McPharland, & Coyne, 1991; Cornsweet, 1990; Klinge, Piggott, Knitter, & O’Donnell, 1986; Gabel & Shindledecker, 1990). Throughout history, progress in the treatment of adult psychopathology has outpaced that of children and adolescents (Cornsweet, 1990; Zimmerman, 1990). Until the late 1800s, disturbed children were often labeled delinquent, and frustrated parents customarily sent them to foster homes and other custodial institutions. Such facilities emphasized work and conformity and the children placed there generally were not expected to become productive, well-functioning members of society.

During the early 1900s, many new hospitals designed for the specialized care of adult psychiatric disorders opened. Twenty years later, the first children’s psychiatric units were established at Bellevue, the Franklin School, Allentown State Hospital, and Kings Park State Hospital. The first all-adolescent units appeared in the late 1930s; however, until the 1950s, convention held that troubled adolescents were generally too aggressive and destructive for inpatient psychiatric care and therefore should be placed in facilities that emphasized strong discipline (Zimmerman, 1990). Services designed specifically for children and adolescents were not widely available until the 1950s, and the practice of treating disturbed children and adolescents in specialized units of hospitals became widespread only in the 1960s (Cornsweet, 1990). Therefore, it should perhaps come as little surprise that evaluations of the effectiveness of therapies designed specifically to treat child psychopathology would be lacking compared with assessments of adult treatment.

Heightened awareness of the impact of mental illness on society and the economy has increased concern about the effectiveness and cost of mental health services (Kazdin, 1990). Mental health care coverage, although increasingly scarce, represents a significant cost to businesses. In the late 1980s, the cost of mental health care was rising at a rate of 27% a year, making it the fastest rising health care cost (Moreland, Fowler, & Honaker, 1994). The US House of Representatives’ Office of Technology Assessment reported in 1986 that 7.5 million children in the United States were experiencing an emotional, behavioral or developmental problem, and estimated that only 20-30% were receiving treatment, while even fewer were receiving treatment that was deemed adequate (cited in Rog, 1992).

Fiscal constraints have often caused policymakers to focus on reducing mental health care costs by reducing services, while minimizing the importance of evaluating outcomes achieved by therapeutic interventions (Davis & Frank, 1992). This is probably penny-wise and pound-foolish (cf. Yokley, Coleman, & Yates, 1990). Failing to provide care may have lasting adverse consequences, not only for the children receiving treatment but also for their families and society at large. Many dysfunctions that present in childhood and adolescence can either continue into adulthood essentially unchanged or metamorphose into adult psychopathology (Kazdin, 1993b; Cornsweet, 1990). If childhood disorders are not addressed, their severity may increase, resulting in greater cost in the long run (Berman & Austad, 1991). Untreated cases of childhood conduct disorder, for example, may cost society a great deal owing to delinquent acts committed in adulthood (Day, Pal, & Goldberg, 1994).

Outcome research will demonstrate to third party payers that treatment for troubled children and adolescents is more cost-effective, in the long run, than no treatment (Vermillion & Pfeiffer, 1993). However, the purpose of conducting outcome research extends beyond evaluating the general question of whether psychotherapy is effective. An obvious goal of outcome research should be to improve the quality and efficacy of treatment in order to reduce suffering in children and adolescents. Detailed studies of outcome will delineate which treatments are most effective for which disorders, and which factors tend to reliably predict favorable versus unfavorable outcome (Kazdin, 1990; Vermillion & Pfeiffer, 1993). Therefore, increased knowledge about outcomes may ultimately influence initial choice of treatment (Kazdin, 1993b). Outcome research serves as an important source of information for the design of prevention programs for at-risk children and adolescents (Kazdin, 1993b). Positive results obtained from outcome studies should also increase public awareness of the value of psychiatric treatment and thereby influence public policy regarding mental health care services.

Findings of Outcome Research on Child and Adolescent Psychotherapy Conducted to Date

An early review of child psychotherapy by Levitt (1957) called into question the efficacy of psychotherapy in much the same way that Eysenck’s controversial review of adult psychotherapy had in 1952. In his evaluation of studies of child psychotherapy outcomes, Levitt (1957) found no evidence to support the assertion that treated children fared substantially better than untreated children. Several criticisms have been leveled against Levitt’s methodology and conclusions. For example, the studies he reviewed included individuals whose ages ranged widely: from preschool through twenty-one years of age (Kazdin, 1990). Also, his untreated group consisted solely of children who were followed up after dropping out of treatment (Weisz, Weiss, & Bononberg, 1992). Levitt apparently failed to consider that the dropouts may have been qualitatively different from the children who remained in treatment, therefore making them inappropriate as controls. Finally, in Levitt’s review, improvement was assessed by therapist ratings alone, without assessments by parents, or the patients themselves (Kazdin, 1990). It is now well known that assessments of improvement can vary widely depending on the source of the evaluation (Achenbach, 1994; Lachar & Kline, 1994; Casey & Berman, 1985).

Most recent meta-analyses¹ have found that treating children and adolescents yields improvement that is unlikely to occur simply as a function of time (Kazdin, 1993a; Casey & Berman, 1985; Weisz, Weiss, Alicke, & Klotz, 1987; Blotcky, Dimperio, & Gossett, 1984; Pfeiffer & Strzelecki, 1990). Recent meta-analyses collectively evaluating hundreds of outcome studies of child and adolescent mental health services have found significant positive effects of psychotherapy for children and adolescents, estimating that children who receive treatment function better than between 76%-81% of non-treated children (Casey & Berman, 1985; Weisz et al., 1987; Weisz et al., 1992). These findings are similar to the results obtained in recent meta-analyses of outcome studies of adult psychotherapy (CritsChristoph, 1992; Shadish, Montgomery, Wilson, Wilson, Bright, & Okwumabua, 1993).

Speculations About Outcomes from Predictor Variables

Several of these meta-analyses have attempted to identify predictor variables which may inform clinicians regarding choice of technique, nature and composition of treatment, expected course of therapy and/or hospitalization, and level of expectation for favorable outcome. These have included age, intelligence, gender, severity of dysfunction, and family functioning. Meta-analysts have reached substantial agreement on the effects of some of these variables, but remain divided on the impact of others.

Researchers appear to have reached a fair degree of consensus regarding the negative implications of organic brain dysfunction and severe “functional” diagnoses, antisocial behavior, and bizarre symptomatology. Not surprisingly, children who present a healthier clinical picture tend to have more favorable outcomes (Kolko, 1992; Blotcky et al., 1984; Pfeiffer & Strzelecki, 1990; Gossett, 1985).

There is also general agreement that healthy family functioning and supportive after-care services lead to significantly better outcomes (Blotcky et al., 1984; Pfeiffer & Strzelecki, 1990; Gossett, 1985). A study by Lewis (1988) found that measures of family functioning at admission were related to later adjustment. It is clear that young people from dysfunctional families require substantial after-care and support after release from residential care (Curry, 1991; Cornsweet, 1990; Pfeiffer, 1989; Gabel & Shindledecker, 1990; Gossett, 1985). Hence, changes in youngsters produced by residential care often must be accompanied by positive change in the family setting (Curry, 1991; Lewis, 1988).

Researchers have not reached considerable agreement on the value of age, gender, and intelligence as predictors of outcome. For example, Casey and Berman (1985) found that studies which included a greater proportion of male children usually produced smaller treatment effects, while Blotcky et al. (1984) found a slight disadvantage for girls, and a review by Pfeiffer and Strzelecki (1990) found no relationship between sex and outcome.

Weisz et al. (1987) argued that age has a significant effect on outcome, with children faring better than adolescents; however, Pfeiffer and Strzelecki (1990) detected only a weak relationship between age at admission to a psychiatric facility and outcome, and Casey and Berman (1985) found that age did not predict outcome. Kolko (1992), on the other hand, found that older age was predictive of poor short-term outcome. It is important to remember that age at admission to a psychiatric facility, and therefore severity of pathology, may differ as a function of several variables, such as parental anxiety about the hospitaHzation of their children or ability of schools to handle problem behaviors (Blotcky et al., 1984).

Intelligence is a variable that is often examined in outcome studies. Both Gossett (1985) and Blotcky et al. (1984) found that higher intelligence correlated with greater improvement, while the review by Pfeiffer and Strzelecki (1990) revealed a more modest relationship between intelligence and outcome, and Casey and Berman (1985) found that intellectual functioning was not reliably related to outcome. These inconsistent results may be a function of the fact that tested IQ itself may be affected by psychopathology, and these effects may differ according to severity of disorder. Tested IQ is also correlated with socioeconomic status. Neither of these variables is generally controlled in studies where IQ scores are considered (Blotcky et al., 1984). It may be more informative for studies to employ IQ as an outcome measure as opposed to focusing on IQ as a predictor of outcome (Zimmerman, 1990).

Correlates of Outcome

Length of hospitalization in a psychiatric facility is a variable that is often considered in studies of inpatient treatment, not as a predictor, but rather as a correlate of outcome. Many children and adolescents improve during inpatient treatment, but an optimal time period for hospitalization has not been clearly established. The operational definition of short-term hospitalization has ranged from 11 days to 89 days, while long-term hospitalization has ranged from 24 days to 179 days. The overlap has meant that what some studies have defined as short-term, others have defined as long-term (Caton, Mayers, & Gralnick, 1990). Therefore, in order to avoid confusion in the interpretation of results from studies of treatment, it is important to consider the researcher’s definition of long versus short-term care.

The meta-analysis by Blotcky et al. (1984) found a positive relationship between length of stay and favorable outcome, with most studies suggesting that positive results required hospitalizations of at least 6 months to one year. Pfeiffer and Strzelecki (1990) found a modestly strong positive relationship between longer length of hospitalization and favorable outcome. It appears that, owing to cost cutting by insurance companies, children are increasingly entitled to only brief hospitalizations, regardless of their diagnosis or the severity of their pathology (Caton et al., 1990; Nurcombe, 1989). The results of some studies seem to suggest that shorter hospitalizations may facilitate smoother, less traumatic reintegrations into the family, school and community (Gossett, 1985; Pfeiffer & Strzelecki, 1990). It may be that a rigorous course of post-treatment outpatient care after brief hospitalization can be as effective as traditional, long-term hospitalization in treating psychopathology and preventing rehospitalization (Gunstad & Sherman, 1991; Berman & Austad, 1991).

Ney, Adam, Hanton, and Brindad (1988) found that an important aspect of length of stay is specification, at admission, of exact length of hospitalization. Ney and colleagues argued that knowledge of the exact length of stay helps to concentrate staff effort and foster the involvement of family members, maintaining the children’s attachment bonds. Unfortunately, their sample was quite small and therefore not necessarily generalizable. Gunstad and Sherman (1991) similarly argued that discharge planning should begin upon admission to a psychiatric inpatient facility.

Treatment characteristics have not been considered as potential correlates of outcome in most studies and meta-analyses (Blotcky et al., 1984). Representing an exception to this generalization, Casey and Berman (1985) compared the efficacy of different types of treatment, including group versus individual therapy, play versus non-play, and parent-treated versus child-treated. They found no significant differences between these groups. In their comparison of behavioral versus non-behavioral therapies, Weisz et al. (1987) found that behavioral treatments were significantly more effective. It should be noted, however, that studies of behavioral therapies have often employed outcome measures that were very similar to activities occurring during treatment (Blotcky et al., 1984). When these therapy-like measures were removed from the evaluation by Weisz et al. (1987), this difference between behavioral and non-behavioral therapies disappeared. In general, comparisons between different psychotherapeutic techniques have yielded small effect sizes² (Kazdin, 1990).

Meta-analysis enables researchers to simultaneously evaluate the findings of a number of studies with different research designs and sample sizes by converting results to a common metric and calculating an overall effect size. Meta-analytic reviews of outcome studies can therefore be instrumental in the identification of common strengths as well as areas of neglect in research (Kazdin, 1993b). Conclusions supported by a combination of many studies may be more reliable than results obtained from a single study.

However, findings of meta-analytic reviews must be interpreted with some degree of caution as a result of the potential repercussions of the process of combining studies. Studies that are reviewed in meta-analyses have often employed different methods of sampling, collecting data, analyzing results, and reporting findings. Differences in outcome that are potentially attributable to differences in sample size, research design, therapeutic technique, and operational definition of improvement may therefore be overlooked (Kazdin, 1993b). Additionally, meta-analysts are often forced to estimate, assume, and infer data that have not been reported, but are nevertheless crucial for the purpose of comparison. Furthermore, meta-analyses generally consider studies that have been published. If there is indeed a publication bias in favor of studies which have obtained significant effects, then meta-analyses have not considered many studies that have failed to produce statistically significant effects.

Shortcomings of Current Outcome Research

Recent evaluations of meta-analytic reviews suggest that meta-analysts may have been overly enthusiastic in their conclusions about the positive effects of treatment (Weisz et al., 1992; Matt, Shadish, Navarro, & Siegle, 1994). An important limitation of many outcome studies that have demonstrated beneficial effects of therapy over no therapy, which has frequently been overlooked, is the fact that the treatment process in the research studies differs significantly from treatment as it is administered in actual clinical practice. Most of the studies evaluated in the aforementioned meta-analyses have included at least one component that makes the study differ from clinical care as it is actually administered. Given important differences between therapy as studie...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• Preface
• 1. Introduction to Treatment Outcome: Historical Perspectives and Current Issues
• 2. Criteria for Selecting Instruments to Assess Treatment Outcomes
• 3. A Comparison of Commonly Used Treatment Measures
• 4. Implementing an Outcome Assessment Project: Logistical, Practical, and Ethical Considerations
• 5. Measuring Outcomes in Residential Treatment with the Devereux Scales of Mental Disorders
• Index