Case Studies: Overview | Pre-Implementation | Formative | Summative
The pre-implementation stage of evaluation consists of both the collection of initial data as well as the formation of a plan of action to guide the evaluation process. It is important to determine the goals and scope of the evaluation at the beginning as this will give direction to the overall evaluation. This section contains examples of pre-implementation procedures and online tools and resources that will assist districts in structuring this stage of their evaluation.
Before implementing a program or policy, districts need to take the time to plan for the evaluation and align the design with the program's central goals. The key steps discussed below are drawn partially from this evaluation guide.
Define the purposes of the evaluation
It is important to isolate expectations for what the evaluation will accomplish. The evaluation should examine progress, assess the accomplishment of intermediate objectives, and provide findings in a timely manner so that program adjustments can be made as required.
- Where To Start is a section of an evaluation toolkit developed by the W. K. Kellogg Foundation that contains information useful during the initial stages of an evaluation. The topics covered are: 1) evaluation purposes; 2) scope of evaluation; 3) type of evaluator; and 4) identifying and involving stakeholders.
- Evaluation Approaches is a section of the W. K. Kellogg toolkit that helps organizations understand the different approaches to evaluation and how these approaches affect the overall evaluation design. Topics covered are: 1) how to choose an approach to your evaluation and 2) major approaches to evaluation.
Understand the evaluation context
Contextual factors surrounding an evaluation may affect the choice of evaluation design, data collection strategies, and the way in which stakeholders use the results. Four areas should be considered: 1) local or school needs; 2) program elements; 3) stakeholder expectations; and 4) availability of evaluation resources and expertise. During this stage of evaluation planning, it may be helpful to conduct initial surveys of community, teacher, and staff perceptions of both the program being implemented and the evaluation itself.
- Worksheet 2 in this evaluation guide is designed to help organize the information that is developed as evaluators respond to questions related to each of these four areas.
- Developing a Survey is a tool provided by Innovation Network that provides advice about developing a survey and modifying existing data collection instruments. Districts should develop internal surveys to assess a variety of factors, including staff and teacher satisfaction, how individual educators are implementing the program, how parents and community members view the reform effort, etc.
Consider the use of an outside consultant
An external consultant can assist at all stages of the evaluation. Areas of expertise include helping to frame the significant questions, identifying or modifying existing instruments, training staff in data collection methods, and developing additional tailored instruments and systems for organizing and monitoring the evaluation process. Operation Public Education provides technical assistance for policy development to support districts interested in implementing and evaluating the OPE framework or its individual components. For information about partnering with OPE, please contact us.
- For information about hiring an external evaluator, the W. K. Kellogg Toolkit has a section on Hiring and Managing an Evaluator. Topics covered include: 1) the evaluator's role; 2) ethical issues and qualifications; 3) finding an evaluator and when to hire an evaluator; 4) making contractual arrangements; 5) working as an evaluation team; and 6) general tips for hiring an evaluator.
- This tool from The American Evaluation Association allows districts to search for evaluation consultants by state. Districts with limited evaluation capacity may want to employ third-party consultants to conduct or design program evaluations.
- For an example of an evaluation plan proposal for working with an outside consultant, see this document developed by Eagle County Schools (ECS). The district created a proposal in 2008 to identify a third-party vendor to provide evaluation support. As part of the proposal, the district highlighted the proposed evaluation plan and data collection processes.
Link the evaluation to the program
The program components of a comprehensive reform should work together toward the ultimate goal of improved student achievement. It is helpful to develop a program logic model to identify the connections between the various components and to ensure that there are no inconsistencies.
- This Logic Model Development Guide from the W. K. Kellogg Foundation helps organizations identify the underlying principles, or "logic model," of their evaluation effort. [Note: The link above goes to a page where you can order the Logic Model Development Guide. The guide is free, but only one per person is available. Fill in the requested information, and the evaluation guide and CD will be mailed to you.]
Identify the significant questions that will shape and guide the evaluation effort
Evaluators should consider questions such as: 1) What does your school want to accomplish overall? 2) What will your school have to do to achieve these goals and objectives? 3) How will your school know that its program is succeeding at accomplishing its goals and objectives? 4) How will evidence be gathered to demonstrate progress toward your school's goals? 5) How will your school interpret the data? 6) How will your school use the evaluation results?
- Evaluation Questions is another section of the W. K. Kellogg toolkit that contains information and tools to help organizations develop useful evaluation questions. The tools are intended to help evaluators identify the specific questions that they want the evaluation to answer. The topics covered are: 1) refining the initial evaluation questions; 2) finding sources that can help in developing evaluation questions; and 3) general tips for developing questions. Within each section, there are a variety of resources including checklists, templates, and planning tools.
Connecticut Charter Schools
For an example of the development of evaluation goals, plans, and significant questions, see this case study of the evaluation of charter schools in Connecticut. The district highlighted central concerns such as:
- To what extent are all students being served?
- To what extent are the stated specific goals and objectives of the schools being met?
- What unique and common shortcomings and barriers to meeting student needs can be identified?
- What successes and shortcomings in the development of the school governance procedures and policies exist or have been developed?
- What are the long-term (positive and negative) effects on students and parents that are associated with attending or sending children to a charter school?
Denver's ProComp
ProComp is a groundbreaking compensation system that links teacher pay to student achievement. Before implementing the program throughout the district, Denver conducted a four-year pilot and evaluation of the initiative in order to determine what works and what does not. Districts with the resources to do so may want to conduct pilots of proposed reforms as part of a more extensive pre-implementation evaluation. Based on Denver's evaluation of the pilot, the district was able to identify a number of issues, recommended actions, and impacts on stakeholders.
The following resources provide additional information on the pre-implementation stage of evaluation.
- Budgeting for an Evaluation is a section of the W. K. Kellogg toolkit that assists organizations in creating an initial evaluation budget. The topics covered are: 1) budgeting FAQs and 2) developing a budget framework.
- The following worksheets and checklists are useful tools for developing a plan of action:
- Plans and Operations: Identifies important considerations when planning evaluations. Questions are presented to prompt evaluators to think proactively about an array of issues relevant to a conceptualization of the evaluation.
- Budget Development: Presents cost categories that should be considered when preparing a budget for an evaluation. Personnel, travel, supplies, communications, printing, equipment, consultants, in-kind services, and overhead are some of the major categories.