Case Studies: Overview | Pre-Implementation | Formative | Summative
Districts that implement reforms require the infrastructure and expertise to measure and assess the results of the changes. This "culture of measurement" should guide districts to collect the appropriate data needed to make a thorough evaluation of the reforms, analyze the data to see whether it meets expected goals, and then report the findings to all relevant stakeholders.
There are three stages of evaluation:
- Pre-Implementation. It is helpful to begin evaluation work as early as possible, both because the initial data can be used to monitor the immediate effects of changes, and because this information can later serve as a baseline for determining effectiveness over time. It is during this stage that districts should decide how the results will be used and how to incorporate the needs of all users in the evaluation design. It is important that the evaluation activities reflect the program's "theory of action," or the intended results of a specific intervention. This will ensure that the data collected is relevant to the identified focus of the evaluation.
- Formative. Formative evaluation activities occur in "real-time" while a reform effort is in progress and can be used to help program directors and district personnel monitor and improve the implementation of the program. This type of basic program monitoring often consists of qualitative data collection, such as feedback from stakeholders, and quantitative data collection, such as the number of teachers and students participating in a given program, how often they participate, and so forth.
- Summative. Summative evaluations assess the overall impact of a reform after a set period of time. Such an evaluation helps isolate the overall effect of the intervention on desired outcomes such as student achievement or teacher retention. Summative evaluations are critically important to many stakeholders, including state legislatures and grant-making foundations, which often require their grantees to demonstrate results before releasing additional funds. This type of evaluation is sometimes completed by a third-party contractor, but the process still requires the participation of district officials.
Evaluation should take place before, during, and after the implementation phase to ensure that program designers have the opportunity to include improvements and changes on an as-needed basis. Effective evaluations can help districts formulate their goals and ask key questions about the implementation of reforms. Since it is vital to ask the right questions to ensure that the needs of all stakeholders are being met, the evaluation process should incorporate feedback from multiple constituencies that can serve as a system of guideposts throughout the implementation process. By engaging in evaluation, program administrators are required to closely analyze data and feedback that otherwise would not be available to help make program adjustments mid-course.
The benefits of conducting program evaluations, highlighted on page 5 of a brief by the Northwest Regional Educational Laboratory (NWREL), include:
- Strengthening the overall reform effort by clearly articulating shared goals and objectives.
- Facilitating informed decision-making about improving the quality of reform efforts.
- Contributing to making constructive changes to enhance program effectiveness.
- Helping to identify and celebrate successes when desired outcomes are achieved.
- Reinforcing the link between school-wide program strategies and student outcomes.
The following recommendations do not address the theoretical issues undergirding evaluation or discuss the more technical concerns associated with choosing particular evaluation models. Rather, they draw from research and publications to provide an overview of some general considerations for districts to keep in mind when choosing an approach for evaluating their school reform efforts. For additional resources, see the websites and tools section.
- Create a "data culture" or "culture of measurement" within the district. This is one of the least tangible yet most vital aspects of successful evaluation efforts. Teachers should not view the collection of data as an additional imposition on their time, but rather as an integral tool that can be used to provide useful feedback within their own classroom as well as the school or district. To develop this culture, districts should incorporate the time and supports necessary to foster data-informed discussions. Additional training will help build teachers' investment and skill in data analysis while also generating important information that can be used to provide ongoing guidance for the implementation process.
- Identify a program vision and align evaluation with desired outcomes. Districts should design a theory of action that sets clear goals for their reform efforts. Evaluation should be aligned with this vision, such that the specific methods of data collection and analysis used respond directly to the questions of interest. Evaluation activities should begin as early as possible and continue throughout the process of implementation.
- Engage all stakeholders in the evaluation planning and implementation process. From the very beginning of an evaluation process, teachers, parents, and community members should be informed of the plan and encouraged to offer feedback based on their personal experiences and observations. Ease of data collection and organization is particularly important to secure the full cooperation of all stakeholders. Data warehousing technology should be used to link information stored in different locations and ensure that data input and interface are easy and user-friendly.
- Build capacity for evaluation. An evaluation team is essential to the successful design and implementation of an evaluation effort. Establishing a core team of staff members to head the evaluation will ensure that plans are carried out efficiently with minimal deviation from the stated goals. The members of the evaluation team should be given sufficient release time to complete these duties. It is useful to designate an evaluation team leader, as well as an advisory group comprised of other stakeholders to whom the core evaluation team can turn for feedback. If necessary, the evaluation team may also want to solicit the help of third-party evaluators. Click here to find a local evaluation consultant.
- Use multiple approaches to data collection to enhance the validity of the data. Districts should collect data throughout the process of reform. Formative assessments can be used during implementation to allow for program improvements mid-course while summative assessments allow districts to evaluate the overall impact of interventions. Employing both quantitative and qualitative methods of data collection during these stages can help explain the findings and enhance the validity of the data.
To see how these recommendations can be implemented, refer to the case study pages.
These publications address best practices in evaluation, as well as how data is being used in a broader sense to drive decision making in school systems across the country.
- No Schools Left Behind
(V. L. Bernhardt, Educational Leadership 60, no. 5, 2003, 26-30)
This article provides an in-depth description of the various types of data that can be used for evaluation and the ways in which the intersections between categories of data can provide a more vivid picture of the school environment. The four types of data identified are demographic, student learning, perceptions, and school processes. - Comprehensive School Reform: An Evaluation Guide for Districts and Schools
(Louis F. Cicchinelli and Zoe Barley, Mid-continent Research for Education and Learning, 1999)
This guide outlines the design and process of evaluating a federal comprehensive school reform initiative, but the advice can be applied to any school conducting a program evaluation. It addresses such topics as implementation, evaluation logistics, types of data, and barriers to gathering and using data. It also includes reproducible worksheets that are incorporated into each chapter. - Achieving with Data: How high-performing school systems use data to improve instruction for elementary students
(Amanda Datnow, Vicki Park, and Priscilla Wohlstetter, Center on Educational Governance, University of Southern California, 2007)
This report, which is based on comprehensive studies of four school systems, emphasizes the need for data-driven decision making in schools and identifies the key strategies of performance-driven school systems. These strategies include (1) building a foundation for data-driven decision making, (2) establishing a culture of data use and continuous improvement, (3) investing in an information management system, (4) selecting the right data, (5) building school capacity for data-driven school decision making, and (6) analyzing and acting on data to improve performance. - Beyond Test Scores: Leading Indicators for Education
(Ellen Foley, et al., Annenberg Institute for School Reform, Brown University)
This paper addresses how to use data to identify indicators that provide early signals of progress toward academic achievement, such as early reading proficiency, enrollment in pre-algebra and algebra, over-age/under-credited students, college admissions test scores, and student attendance and suspensions. The paper discusses the role of the central office in providing data-based support to target assistance to individual students based on these indicators. There is extensive discussion of data warehousing technology, summative and formative assessments, data input and interface, and designing the supports necessary to foster data-informed discussions. The appendix contains question templates for interviewing a variety of individuals within the district. - Taking Data to New Depths
(Nancy Love, JSD 25, no.4, Fall 2004, 22-26)
This article contains strategies for examining multiple levels of data in order to draw sound inferences. The article focuses on the improvement of mathematics and science teaching, but many of the suggestions and considerations are relevant to more general use of data-based evaluation. There is a step-by-step process described for the use of data to achieve student learning goals. One key focus is on training teachers and administrators to lead collaborative data-based improvement programs. - Making Sense of Data-Driven Decision Making in Education: Evidence from Recent RAND Research
(Julie A. Marsh, John F. Pane, and Laura S. Hamilton, RAND Corporation, 2006)
This paper draws on four studies on the use of data in educational contexts. It provides a broad overview of data-driven decision making (DDDM) and its effect on student outcomes. The paper examines both the kinds of questions and types of data that administrators use to make decisions. There is also an overview of the factors, such as accessibility and quality of data, which influence the use of data for decision making. The paper concludes with policy and practice implications, as well as areas for further research. - Evaluating Whole-School Reform Efforts: A Guide for District and School Staff
(Kim Yap, et al., Northwest Regional Educational Laboratory, 2000)
Educators at the NWREL created a guide for district and school staff to assist them in choosing an evaluation methodology to assess school reform efforts. The guide raises a number of key considerations and compares the relative advantages and disadvantages of different kinds of evaluations, such as implementation (formative) and impact (summative).
The following resources contain additional ideas and information on how to evaluate school reform initiatives. They provide access to ready-to-use materials aligned to the above recommendations and also suggest various external organizations that districts could work with to develop a strategy.
Ready-to-use materials
- Developing a survey is a free tool provided by the Annenberg Institute that offers advice for developing targeted surveys and modifying existing data collection instruments.
- The Evaluation Center at Western Michigan University aims to advance the development, validation, and utilization of evaluation. The Center currently oversees a number of program evaluations and has published many evaluation reports. In addition to other resources, they have created a group of evaluation checklists to aid in all stages of the evaluation process.
- Innovation Network is a nonprofit organization that provides consulting, training, and online resources to evaluate and improve programs. The organization also offers free, publicly-accessible tools and resources such as a self-assessment survey and an evaluation plan builder to improve an organization's ability to plan and implement an evaluation.
External organizations
- The American Evaluation Association is an international professional organization of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Their website contains links to a variety of evaluation publications and other resources, including a tool to search for evaluation consultants by state.
- The National Center for Education Evaluation and Regional Assistance, part of the Institute of Education Sciences (IES) of the U.S. Department of Education, conducts unbiased large-scale evaluations of education programs and practices supported by federal funds, provides research-based technical assistance to educators and policy makers; and supports the synthesis and the widespread dissemination of the results of research and evaluation throughout the United States. Also see What Works Clearinghouse, the IES repository for scientific evidence of "what works" in education.
- The RAND Corporation is a nonprofit, nonpartisan research institution that aims to inform policy and decision making in both the public and private sectors by providing sound research and analysis. RAND strives to provide clients with the necessary knowledge, recommendations, and solutions to enact enduring reforms. RAND engages in research in many policy areas including K-12 education. The organization has conducted studies of the issues of assessments, accountability, school reform, and teacher quality among many others.
- Evaluation: A Systematic Approach
(Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman, SAGE Publications, 2004)
This text is considered one of the seminal pieces on the design, implementation and appraisal of programs through the use of evaluation. The authors include a number of evaluation techniques, guidelines for tailoring evaluations to fit specific programmatic contexts, and examples of successful evaluations. Topics covered include framing evaluation questions, uncovering program theory, studying implementation, designing impact assessments, assessing program costs and benefits, and understanding the politics of evaluating. - A Practical Guide to Program Evaluation Planning: Theory and Case Examples
(Debra A. Holden and Marc A. Zimmerman, SAGE Publications, 2009)
This book provides a step-by-step process to guide evaluators in planning a comprehensive, yet feasible, program evaluation - from start to design - within any context…the case examples presented in this book provide evaluators with the conceptual tools and a framework for thinking about the pre-evaluation tasks that may be necessary for a successful, useful, and informative evaluation. - Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness
(Huey-Tsyh Chen, SAGE Publications, 2005)
This book introduces students to the real world of evaluation, focusing on issues that arise in professional practice. Author Huey T. Chen concentrates on the steps vital to program evaluation: systematically identifying stakeholder needs, selecting evaluation options best suited to particular needs and reconciling any necessary tradeoffs, and turning the decisions into action. Using illustrative examples from the field, the book details the methods and procedures involved in analyzing and identifying stakeholder expectations and selecting an evaluation strategy. The principles and strategies presented are reinforced with theoretical justification. This provides students and professional evaluators the knowledge and flexibility to design suitable evaluations and follow them through from start to finish.