The Center for Comprehensive School Reform and Improvement February 2005 Newsletter

The U.S. Department of Education has released the Longitudinal Assessment of Comprehensive School Reform Program Implementation and Outcomes: First-Year Report. This first report of the planned five-year, longitudinal study of the Comprehensive School Reform (CSR) program found that CSR schools were more likely than non-CSR schools to adopt whole-school reform models, offer professional development, and use research-based improvement strategies. The analysis also revealed that states are likely to award CSR funds to high-poverty, low-achieving schools. This newsletter summarizes the central findings of the report; the complete text is available at:
http://www.ed.gov/rschstat/eval/other/lacio/lacio-final.pdf.

Introduction

CSR is a federally funded program that encourages schools to marshal their resources into a single, schoolwide, “comprehensive” reform program rather then engage in fragmented improvement efforts. The initiative began in 1998 with a $145 million Congressional appropriation, and over the past five years more than 5,000 schools have received CSR funds. Under the program, each school receives a grant of at least $50,000 a year renewable for up to three years.

The program aims to raise student achievement by requiring schools to implement comprehensive reforms that are grounded in scientifically based research and effective practices. Whether they use a nationally available approach or develop their program locally, CSR schools must coherently integrate 11 federally identified components of reform.

Evaluation Design

The LACIO study does not focus on the implementation of whole-school reform “models” but instead will examine the degree to which reforming schools implement the 11 required CSR components. The study focuses on the following four questions:

1. How are CSR funds being targeted?

2. How is comprehensive school reform implemented?

3. What is the relationship between CSR implementation and student achievement?

4. How do conditions at the state and district level influence the implementation of comprehensive reform programs?

Taken together, the answers to these questions will provide “information about the extent to which CSR helps low-performing schools change practices and improve outcomes with appropriate support, including support in implementing research-based practices and organizational structures” (Tushnet, Flaherty, & Smith, 2004, p. 7).

The study uses a quasi-experimental design and over time will analyze both student achievement data and survey and interview data from a sample of 800 schools. Of these, 400 will be CSR schools, randomly chosen from all schools that were funded by CSR in 2002. The other 400 will be non-CSR schools that are similar to the CSR schools in demographics and student achievement.

LACIO will compare student achievement by analyzing data housed in the National Longitudinal School-Level State Assessment Score Database, maintained by the American Institutes for Research. It will survey school reform activities at various times over five years by measuring actual behaviors rather than attitudes and expectations. (For the first-year report, schools were surveyed in spring 2003 at the end of their first year of CSR implementation.) Evaluators also will produce case studies of a small number of CSR and non-CSR schools in their district and state context as well as analyze what happens to schools after their CSR funding ends.

WestEd and the COSMOS Corporation are conducting the research. Assistance is being provided by Duerr Evaluation Resources.

First-Year Findings

The first-year evaluation report contains a number of findings. CSR funds have been awarded to schools with high numbers of poor students and low student achievement. Nearly half of the CSR schools in the study had both poverty rates and minority enrollment of 75 percent or above, and CSR schools were more likely than the schools in the comparison group to be identified by their state as low performing (46 percent) than were the non-CSR schools (28 percent).

Researchers found that while both CSR and non-CSR schools are engaged in reform activity, CSR schools are more likely to do the following:

• Employ reform models that have been used elsewhere. CSR schools were more likely to adopt an externally developed reform model (32 percent of CSR schools versus 6 percent of non-CSR schools) when they developed their comprehensive school reform design. They also reported more often that their school improvement plan was influenced by the specifications of a reform design. Success for All, Renaissance Learning, Co-nect, America’s Choice, and Direct Instruction were the models most frequently named by CSR schools as being used in their schools.
• Offer more continuous professional development. CSR schools were more likely to dedicate more than 10 days to teacher professional development (56 percent versus 39 percent) and use outside trainers (85 percent versus 57 percent). Also, teachers in CSR schools reported participating in more days of professional development than they did the previous year and said that their training on those professional development days was more focused on reform issues, such as instructional strategies.
• Garner school and staff support for reform. CSR schools were more likely to ask the entire teaching staff to vote on the decision to implement a reform model (82 percent versus 55 percent). CSR schools also were more likely to report that decisions regarding reform were made at the school level instead of by their state or school district. State or district requirements were more likely to influence reform at non-CSR schools (60 percent) than at CSR schools (31 percent).

In some areas, CSR schools and non-CSR schools did not differ significantly in their practices or behaviors. Most notably, CSR-funded schools did not coordinate financial resources to any greater extent than did non-CSR schools. “We are following the money to know exactly what a school does with it,” says Naida Tushnet, director of WestEd’s Evaluation Research Program and primary author of the report. “Coordination of financial resources is a big problem for all schools. Partly, it’s because each source of state or federal funds has its own accountability methods. Although integration is allowable, schools worry that if they do it, they won’t be in compliance.”

Implications

Results from the first year of LACIO suggest that the road to reform is slow and rocky. CSR funds did not cause any dramatic turnarounds. “Schools have to be willing and want to stay the course. They will not see any easy wins at first,” says Tushnet. “What they will see if they reform is the building of teacher communities, more conversations about teaching and learning. [Reform] starts by getting teachers focused. One example of this is the increased number of professional development opportunities in CSR schools.”

Perhaps the most promising LACIO finding is that CSR schools are more likely to be engaged in research-based reforms. CSR schools are more likely to adopt an externally developed reform model as well as cite specific reform models in their comprehensive plans. “Planning coherent research-based reform is very helpful for schools,” says Tushnet. This finding, says Tushnet, suggests that CSR schools are likely to have student achievement gains because they are employing reforms that have already been shown to increase student learning.

What’s Next

Future reports from LACIO will shed light on how states and districts build capacity in struggling schools and how they help schools after their CSR funding ends. The relationship between reform and student achievement also will be examined to determine what, if any, connection exists between reform initiatives and increased student achievement. “We will be able to tease out what circumstances lead to higher student achievement,” says Tushnet. “No reports have tried to document this on a national scale.”

References

Tushnet, N. C., Flaherty, J., Jr., & Smith, A. (2004). Longitudinal assessment of Comprehensive School Reform program implementation and outcomes: First-year report. Washington, DC: U.S. Department of Education.