Evaluating Systems of Care

The Comprehensive Community Mental Health Services for Children and Their Families Program. A Special Issue of children's Services: Social Policy, Research, and Practice

E. Wayne Holden, Ana Maria Brannan

• Publisher: Taylor & Francis
• Year: 2014
• Print ISBN: 9781138401181
• eBook ISBN: 9781135587178
• Edition: 1
• Topic: Psychology
• Subtopic: Developmental Psychology
• Index: Psychology

This issue consists of five articles profiling different aspects of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Over the last eight years, this program provided grants to develop community-based systems of care for children with serious emotional disturbance and their families. A comprehensive, multilevel evaluation has been conducted that has provided information to local grantee communities and the federal government on the implementation and effectiveness of systems of care. Touching on a range of questions that the evaluation is designed to address, the articles in this special issue provide more general information on the system-of-care approach to addressing children's mental health problems.

Overview of the National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program and Summary of Current Findings

Brigitte Manteuffel and Robert L. Stephens
ORC Macro Atlanta, GA

Requests for reprints should be sent to Brigitte Manteuffel, ORC Macro, 3 Corporate Square, Suite 370, Atlanta, GA 30329. E-mail: [email protected]

Rolando Santiago
Center for Mental Health Services Substance Abuse and Mental Health Services Administration Washington, DC

In this article we present an overview of descriptive and longitudinal outcome data collected by the national evaluation of the Comprehensive Community Mental Health for Children and Their Families Program. This program, supported by the federal Center for Mental Health Services at the Substance Abuse Mental Health Services Administration, has established systems of care for mental health services in 67 communities throughout the United States. Among the 22 communities receiving grants in 1993 and 1994, descriptive information was collected on 44,640 children who received services. Longitudinal outcome study enrollment included 18,884 children, with data collected on 2,580 children who continued in services through 24 months. The average age of children served was 12.1 years; 61.9% were boys, 54.7% were White, and 60.3% had annual household incomes below $15,000. Primary diagnoses included conduct-related disorders (29.3%), attention deficit hyperactivity disorder (13.6%), and depression or dysthymia (26%). Changes in children’s behaviors and functioning were examined to 2 years in services; 44.6% of children exhibited clinically significant improvements in behavioral and emotional symptoms at 2 years, and 49.5% showed similar changes in functional impairment.

The first Surgeon General’s Report on Mental Health (U.S. Department of Health and Human Services, 1999) was a landmark event detailing the significant resources that have been devoted to the development of mental health services over the past several decades. The report concluded that approximately 20% or 1 in 5 children are affected at any one point in time by a mental health disorder. In 1993, an estimated 9 million to 13 million children and adolescents in the United States (14% to 20%) suffered from a mental health disorder diagnosable under the Diagnostic and Statistical Manual of Mental Disorders (4th ed. [DSM-IV]; American Psychiatric Association, 1994), and about 3.5 million children (3% to 5%) experienced serious emotional disturbance with accompanying functional impairment. Considerable progress has been made in addressing the mental health needs of children; yet children with these needs and their families continue to be underserved, and many questions remain regarding the development of effective community-based mental health services for youth. These questions span a wide range of issues including such areas as the accurate identification of children and families needing services, the integration of a wide array of interventions into community mental health services, efficient financing of services, and continuing efforts to reduce the stigma associated with mental health disorders among children and their families. As Farmer (2000) identified, understanding system level outcomes, as well as individual level outcomes experienced by children and families, is particularly challenging.

The Comprehensive Community Mental Health Services for Children and Their Families Program, administered by the federal Center for Mental Health Services within the Substance Abuse and Mental Health Services Administration, represents the largest federal investment ever to develop community-based mental health services for children and their families. This program, which began to support communities in 1993, provides grants to states, communities, territories, and Native American tribes to improve and expand their systems of care to meet the needs of children and adolescents with serious emotional disturbances and their families. In total, 67 grants have been awarded to communities in 43 states. In 1993 and 1994,5-year grants were made to 22 communities; in 1997,9 communities were funded; in 1998,14 communities were funded; and in 1999 and 2000,22 more communities were funded.

The system-of-care program theory (Stroul & Friedman, 1986) asserts that to serve children with serious emotional disturbance, service delivery systems need to offer a wide array of accessible, community-based service options that center on children’s individual needs, include the family in treatment planning and delivery, and provide services in a culturally competent manner. An emphasis is placed on serving children in the least restrictive setting. In addition, because many children with serious emotional disturbance use a variety of services and have contact with several child-serving agencies, service coordination and interagency collaboration are critical. The system-of-care approach holds that if services are provided in this manner, outcomes for children and families will be better than can be achieved in traditional service delivery systems. In the system-of-care program theory model, agencies in various child-serving sectors, suchas education, juvenile justice, mental health, and child welfare, work together to provide the wide array of services needed by children with serious emotional disturbance and their families. Built upon the Child and Adolescent Service System Program philosophy that calls for services to be child centered, family focused, community based, and culturally competent, the model emphasizes the need to (a) broaden the range of nonresidential community-based services, (b) strengthen case planning across child-serving sectors, and (c) increase case management capacity to ensure that services work together across sectors and providers.

National Evaluation

To examine the implementation of system-of-care theory, as described by the model, the national evaluation of this program was designed to answer the following overarching questions:

• To what extent do systems of care develop and improve over time?
• What services do children and families receive, what service utilization patterns do they experience, and what are the costs of those services?
• What are the characteristics of the children and families served by systems of care?
• To what extent do client outcomes improve over time?
• To what extent do children and families experience service delivery in keeping with the system-of-care program model?
• To what extent can improvements in children’s behavior and functioning be associated with a system-of-care approach?

To address these research questions, a complex, multiple component design was used. Five study components comprised the core of the evaluation: (a) a collection of descriptive data about each child and family served by the program; (b) a longitudinal outcome study of a subset of children and families receiving services; (c) an assessment at the system level to examine the development of the system of care in each community over the course of the funding period; (d) an assessment of the types of services received by program participants, utilization patterns, and associated costs through examination of data recorded in management information systems; and (e) a study comparing three communities with system-of-care funding to three matched communities without funding.

Information presented here is limited to descriptive and longitudinal outcome data from communities funded in 1993 and 1994. Descriptive information including demographics and clinical or service histories provides a comprehensive picture of the characteristics of children and families served across system-of-care grant communities. The outcome study examines changes in child clinical and functional status and family life over time among those receiving system-of-care services. Collecting data in each grantee community enhances the understanding of the unique circumstances found within a particular community that may impact outcomes for children. Examining these features along with changes children and families experience as a result of receiving services in a system of care can contribute to the development of stronger programs in all communities.

Method

Study participants were children from birth to age 22 with serious emotional disturbances and their families receiving services supported by the Comprehensive Community Mental Health Services for Children and Their Families Program in 22 communities that received 5-year program grants in 1993 and 1994. These communities were located in 16 states, ranged from rural to urban in setting, and had diverse ethnic populations. Children participating in the study entered services between 1994 and 1999. Descriptive data were collected on any children receiving services, whereas children between ages 5 and 17.5 years at entry into services, who did not have siblings in the evaluation and had caregivers who consented to their own and their children’s (for youth 11 and older) participation, were followed longitudinally in the outcome study. A caregiver was defined as the person who had primary caretaking responsibility during the assessment period.

The outcome study followed a simple pretest-posttest replacement design, with data collected at intake into services, at 6 months, 1 year, and annually thereafter up to 36 months for as long as children remained in services. The length of time children remained in the outcome study was also influenced by timing of entry into services. That is, children entering services in the second year of funding could be followed longer than children entering services in the fourth year of funding. Additional data were collected as families exited services. When children and families exited services (and therefore the evaluation), or were lost to follow-up, they were replaced with a new family entering services. The number of children enrolled in the outcome study varied across grant communities for reasons that included the size of the community and the proposed program. In total, 21 of the 22 1993-1994 grant communities participated in the outcome study; participation of one community was hindered by the large geographical area served and limited available local staff for the evaluation. Data aggregated from these communities were used in this study. Outcome study enrollment included 18,884 children; at follow-up, data were collected on 8,065 children at 6 months, 5,995 children at 12 months, 2,580 children at 24 months, and 644 children at 36 months. Given the small number of children for whom data were available at 36 months, longitudinal analyses were limited to outcomes at 24 months.

Participants

Data were obtained on at least one evaluation instrument for 44,640 children enrolled in system-of-care programs across the 22 communities. Due to missing data and variations in descriptive data collection across communities, the number of children for whom data were available varied. Among 40,322 children for whom data were available, ages ranged from less than 1 year to 23 years, with a mean of 12.1 years (SD = 4.05); 21.3% were under age 9, 42.8% were between 9 and 14 years old, and 35.9% were 15 years and older. About two thirds of the children (61.9%) were boys (N = 40,428). Slightly more than half were White (54.7%); 24.5% were Hispanic; 14.7% were African American; 3.4% were Asian, native Hawaiian, or Pacific Islander; and 1.8% were Native American or Alaskan Native (N = 38,201).

Additional descriptive information was obtained for more restricted samples. Among 9,855 children, most (47.8%) were in the custody of their biological mothers only; 25.9% were in the custody of two parents (biological or biological and step), 5.1% were in their fathers’ custody, 7.1% were in the custody of guardians (e.g., adoptive parents, foster parents, aunts or uncles, grandparents), 10.3% were wards of the state, and 3.9% had some other custody arrangement. Family income data for 8,142 children indicated that 60.4% of families had annual incomes below $15,000, 18.2% had annual incomes between $15,000 and $24,999, and 21.4% had incomes of $25,000 or higher.

Measures

Descriptive data. Child and family descriptive information included demographics, child and family risk factors, referral source, family income, custody status, mental health service use history, diagnoses, and presenting problems.

Child Behavior Checklist and Youth Self-Report. Behavioral and emotional problems were assessed using the Child Behavior Checklist (CBCL; Achenbach, 1991a), and the Youth Self-Report (YSR; completed by children 11 years of age and older; Achenbach, 1991b), both widely used measures of children’s behavioral and emotional problems in the field of children’s mental health services. The CBCL consists of 118 problem behavior items classified into internalizing or externalizing behaviors. The YSR uses 112 items to assess the same behaviors from the youth’s perspective. Internalizing, externalizing, and total problem T scores can be calculated, with scores above 63 falling in the clinical range.

Child and Adolescent Functional Assessment Scale. Level of functioning was assessed using the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990). The CAFAS was designed to be used to assess the degree of psychosocial functioning of children or adolescents ranging in age from 5 to 17.5 across eight domains: (a) role performance in school, (b) role performance at home, (c) role performance in the community, (d) behavior toward others, (e) moods and emotions, (f) self-harmful behaviors, (g) substance abuse, and (h) thinking. Scores in each of these domains and a total score based on the eight subscales are calculated, with higher scores indicating greater functional impairment.

Residential Li...

Table of contents

• Cover Page
• Children’s Services: Social Policy, Research, and Practice
• Background for the Special Issue
• Overview of the National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program and Summary of Current Findings
• The Impact of Managed Care on Systems of Care That Serve Children With Serious Emotional Disturbances and Their Families
• System-of-Care Assessment: Cross-Site Comparison of Findings
• Policy Implications of the National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
• Commentary: The National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program: A Commentary