Case Studies: Overview | Pre-Implementation | Formative | Summative
Summative evaluation, sometimes referred to as impact evaluation, assesses overall program achievement by determining the degree to which a program has accomplished predetermined goals. Whereas a formative evaluation looks at individual components while they are being implemented, the summative evaluation incorporates all data to reach a conclusion about the overall value of the implemented program. This kind of evaluation is crucially important to many stakeholders, including state legislatures and grant-making foundations, which often require their grantees to demonstrate results before releasing more funds. The outcomes that are assessed most often by a summative evaluation are whether or not a specific program improves student achievement - on metrics such as state and local student achievement tests - and what differences it has made in the school by answering questions such as:
- Has there been a change in school policy or practice?
- Have instructional strategies been strengthened?
- Has student achievement improved?
- What was the most effective element of the program?
- Was it cost effective?
There are various evaluation models that can be used to assess the overall effectiveness and impact of a program. Several of the most commonly used models and basic implementation steps for each are described below. The following information was drawn from this guide to evaluating whole-school reform efforts.
Pretest-Posttest Model
This model assumes that an intervention occurs between pretest and posttest and that any difference that is detected between the two points in time can be attributed to the intervention. The model can include repeated measures, such as measuring teaching practice and student achievement regularly at predetermined intervals. To use this model, evaluators should follow these steps:
- Decide what outcomes you want to assess.
- Select or develop instruments to collect the pertinent data.
- Decide whether sampling is desired.
- Administer the instruments to target groups at pretest time (for example, the beginning of the school year.)
- Administer the instruments at posttest time (for example, the end of the school year.)
- Analyze and interpret the evaluation data.
- Report findings to stakeholder groups.
- Use evaluation data for accountability and program improvement.
Comparison Group Model
This model provides an expectation of program outcomes based on a comparable group. The comparison group provides a basis for determining what might be expected to occur in the absence of the intervention. The comparison group should be similar or equivalent to the intervention group in all relevant respects. Some of the important factors to match include current achievement level, socioeconomic and related achievement factors, school locale, and size. With all other things equal, the assumption is that any detected difference between the two groups is an impact of the intervention. To use this model, evaluators should follow these steps:
- Decide what outcomes you want to assess.
- Select or develop instruments to collect the pertinent data.
- Identify and select a comparison group.
- Decide whether sampling is desired.
- Administer the instruments to both project and comparison groups.
- Analyze and interpret the evaluation data.
- Report findings to stakeholder groups.
- Use evaluation data for accountability and program improvement.
Regression Model
Using a statistical procedure called regression analysis, this model projects the outcome had there been no intervention. This prediction takes into account the range of factors - such as demographic data - that may have had an influence on the outcomes. Other things being equal, the difference between actual outcomes and predicted outcomes is then assumed to be a result of the intervention. To use this model, evaluators should follow these steps:
- Decide what outcomes you want to assess.
- Select or develop instruments to collect the pertinent data.
- Identify and obtain data needed to develop a regression equation.
- Develop a regression equation to predict outcomes.
- Decide whether sampling is desired.
- Administer the instruments to target groups.
- Analyze and interpret the evaluation data.
- Report findings to stakeholder groups.
- Use evaluation data for accountability and program improvement.
Control Group Model
This is an experimental design that requires random assignment of students to the intervention and control groups. Random assignment ensures the equivalency of the groups in all pertinent respects other than the intervention itself. Any difference in outcome between the two groups can thus be attributed to the program or intervention. To use this model, evaluators should follow these steps:
- Decide what outcomes you want to assess.
- Select or develop instruments to collect the pertinent data.
- Set up a control group through random assignment of students.
- Decide whether sampling is desired.
- Administer the instruments to both project and control groups.
- Analyze and interpret the evaluation data.
- Report findings to stakeholder groups.
- Use evaluation data for accountability and program improvement.
For a comparison of the advantages and disadvantages of each of these four models, see page 62 of the guide to evaluating whole-school reform efforts.
QComp
This external summative evaluation of the Quality Compensation for Teachers program in Minnesota was designed to assess whether the program worked as intended and to evaluate the preliminary impact of the program on various outcomes. The evaluation report examines data at the district, school, teacher, and student levels, and it describes the research methods that were used in the evaluation. Among various other criteria, the evaluators looked for improved student achievement and efficiency in the investment of resources.
The following resources provide additional information on summative evaluation.
- The Evaluation Plan is a collection of tools provided by the W. K. Kellogg Foundation that teaches organizations about the major components of an evaluation plan. The tool describes the eight components that should be included in the plan and includes information on the following topics: 1) data collection; 2) data analysis; 3) project-level evaluation activities; 4) reporting; and 5) using evaluation findings. Within each topic, there are examples and additional resources that organizations can use.
- This brief discusses various types of data, how they interact, and the ways in which they can be interpreted. The author places data into four categories: 1) demographic; 2) student learning; 3) perceptions; and 4) school processes. Based on these categories, the brief discusses what kinds of data are important for assessing school improvement and how educators can best organize the data for easy access and analysis.