The Promise and Challenge of Evaluating Systemic Reform in an Urban District

Source: Annenberg Foundation

Date: 2004

Abstract:
In Philadelphia, Pennsylvania, the Annenberg Challenge initiative, known as Children Achieving, set the ambitious goal of having every student achieve proficiency in three core subject areas—math, reading, and science—by 2008. With the support of $50 million from the Annenberg Foundation and a $100-million double match provided by Philadelphia businesses and local foundations, the School District of Philadelphia set out to achieve this goal by designing and implementing ambitious reforms in almost all aspects of its work, and, in the words of its fervent superintendent, David Hornbeck, to do it “all at once.” In this chapter, we describe what happened in Philadelphia in the late 1990s and what we learned about school reform during five years of studying the Children Achieving initiative. Drawing on both quantitative and qualitative data collected during the evaluation of Children Achieving, we examine its theory of action, its implementation, its successes and disappointments. We also describe our experience as evaluators, observing the initiative at close range and providing formative feedback. We argue that the initiative demonstrated some promising early gains in achievement. However, serious flaws in design and execution and inadequate attention to the Philadelphia context ultimately limited its impact and brought it to an end. In particular, we conclude that the policy dictum that everything had to be done at once, as well as poor sequencing of actions, failure to win teacher support for the reforms, and the emphasis on raising standardized-test scores led to uneven, often superficial, implementation. And we conclude that these flaws affected the evaluation by curtailing opportunities to provide candid, timely recommendations for midcourse corrections.

Link: The Promise and Challenge of Evaluating Systemic Reform in an Urban District

Keyword: Evaluation Partnership Strategy

Region: Northern America