Case Studies: Overview | Pre-Implementation | Formative | Summative
Formative evaluation, sometimes referred to as implementation evaluation, generates information used to guide decision-making about a program or reform's desirability, feasibility, fidelity to stated goals, and soundness in producing desired results. Unlike a summative evaluation that judges the ultimate success of the entire program, a formative evaluation is an ongoing process throughout the implementation stage that provides administrators with data in "real-time" that can assist in improving and redirecting the implementation effort mid-course. Formative evaluation should provide a snapshot of program implementation in action and explore whether the program is on track to accomplishing its intended goals by answering questions such as:
- What is working?
- What should be improved?
- How should it be changed?
- Which interventions are being used?
The following steps should be used to direct formative evaluation practices. They are drawn partially from this guide to evaluating whole-school reform efforts.
Decide what data to collect
Based on the research questions developed during the pre-implementation phase, evaluators should decide what data will be most useful and relevant to each target audience.
- This brief discusses various types of data, how they interact, and the ways in which they can be interpreted. The author places data into four categories: 1) demographic; 2) student learning; 3) perceptions; and 4) school processes. Based on these categories, the brief discusses what kinds of data are important for assessing school improvement and how educators can best organize the data for easy access and analysis.
Create specific questions to use for collecting data
Unlike the general evaluation questions developed in the pre-implementation phase, these questions should reflect both the desired data and include the target group from whom the data will be collected. Questions should be clear, specific, pertinent to essential aspects or components of program implementation, and focused on a manageable set of issues.
- Evaluation Questions is a section of the W. K. Kellogg toolkit that contains information and tools to help organizations develop useful evaluation questions. These tools are intended to help evaluators identify the specific questions that they want the evaluation to answer. Topics covered include: 1) refining the initial evaluation questions; 2) finding sources that can help in developing evaluation questions; and 3) general tips for developing questions. Within each section, there are a variety of resources including checklists, templates, and planning tools.
Determine what data-collection tools to use
Evaluators should consider both preexisting instruments that can be modified to meet their specific goals as well as developing new ones as needed. Examples of data collection methods appropriate for formative assessments include surveys, interviews, observations, and review of documents and other artifacts.
- Developing a Survey is a tool provided by the Innovation Network that provides advice about developing a survey and modifying existing data collection instruments. Districts should develop internal surveys to assess a variety of factors, including staff and teacher satisfaction, how individual educators are implementing the program, how parents and community members view the reform effort, etc.
Collect and organize the data
In addition to the data collection tools described above, evaluators should conduct meetings with key stakeholders to gather feedback and perceptions about the program. Evaluators should also consider the use of data warehousing technology or other software that will assist in organizing and managing the data. It is important that the input of data and use of the system are relatively simple so that usage is not limited to only a small number of trained individuals.
- This checklist for gathering feedback from workshops with key stakeholders lists actions that evaluators should take before, during, and after a feedback workshop. Feedback workshops help stakeholders and evaluators: 1) ensure consistency between the evaluation, stakeholder values, and program plans; 2) increase understanding of the evaluation and utility of the findings; 3) improve the accuracy and utility of the evaluation report; and 4) review and refine evaluation plans.
Analyze and interpret the data
While the analysis method chosen depends on both the type of information and the purpose of the evaluation, the results should provide either qualitative or quantitative summaries that reflect the overall evaluation questions. It is often helpful to organize the data separately for each research question as this makes it easier to assess individual components.
Report the data
Evaluation findings should be placed into a written report that addresses the concerns of key stakeholders. This report should include a summary of the results as well as instructions for interpreting the data. These results should be shared both internally and within the broader community. Modified versions of the report should be created as needed to address the concerns of different audiences.
Using data to make program improvements
One of the primary goals of a formative evaluation is to provide information in a timely manner, so that it can be used to impact the ongoing implementation of a particular program. As such, the evaluation should identify any elements of the original plan that are not working as intended, and feedback from teachers and other stakeholders should help determine ways to improve both the program and its implementation.
Oakland Unified School District (OUSD) Small Schools Initiative Evaluation
While this evaluation includes both formative and summative components, the assessment of qualitative data during the formative evaluation is a useful example for districts to consider. OUSD first identified their research goals and then devised a comprehensive strategy to investigate the success of the Small Schools Initiative. Qualitative methods were used to develop case studies of seven new small schools that had shown strong value-added growth for students. Evaluators examined the schools' design features, developmental history, instructional characteristics, and capacity.
ProComp
ProComp is a groundbreaking compensation system that links teacher pay to student achievement. This formative evaluation conducted during the first year of implementation illustrates the mixed-methods of evaluation that should be employed in order to collect the most comprehensive data and qualitative feedback. The evaluation report details the assessment of attitudes and observations of principals and teachers during the initial implementation. The collection of qualitative data relies heavily on the weighted analysis of responses to surveys.
The following resources provide additional information on formative evaluation.
- The Evaluation Plan is a collection of tools provided by the W. K. Kellogg Foundation that teaches organizations about the major components of an evaluation plan. The tool describes the eight components that should be included in the plan and includes information on the following topics: 1) data collection; 2) data analysis; 3) project-level evaluation activities; 4) reporting; and 5) using evaluation findings. Within each topic, there are examples and additional resources that organizations can use.
- These evaluation checklists developed by The Evaluation Center at Western Michigan University provide guidance and strategies for all stages of the evaluation process.