There are essentially four major types or categories of program evaluation—implementation, outcome, impact, and cost studies—with various subtypes within each. Each type of program evaluation is defined and discussed in this section and is followed by four tables that detail the differences in the subtypes of studies and their associated practical considerations, including cost and levels of rigor.
1. Implementation Studies: An implementation study documents program operation or compares it against goals, across locations, or overtime. It describes and analyzes “what happened and why” in the design, implementation, administration, and operation of programs and is generally used to determine whether a program is being carried out in a manner consistent with its goals, design, or other planned aspects. Implementation analyses can serve as stand-alone studies, especially to document new program processes not yet studied. Implementation studies, as part of more comprehensive evaluations, may also include outcome, impact, and/or cost studies. Implementation studies provide context for other or subsequent evaluation findings and results and make the findings or results interpretable and useful for the programs, services, or interventions studied.
2. Outcome Studies: An outcome study compares individual outcomes against goals, across programs or locations, or overtime. Outcome studies differ from impact studies in one key area of comparative data analysis. Essentially, outcome studies determine if programs achieve the desired results or assess the effectiveness of programs to produce change. Nevertheless, outcomes are often thought (by program staff, not program evaluators) to indicate measurable change or “impact” when outcomes are compared over time or across comparable programs.
3. Impact Studies: An impact study estimates the difference in individual outcomes attributable to a specific program or policy. Impact studies determine whether programs or policies measure the intended impacts—that is, the program causes the outcome differences that it is designed to influence. If the purpose of an evaluation is to determine whether an occupational training program has the desired impacts on the employment and earnings of the individuals it serves, an impact study is the ideal type of evaluation to choose. Determining the best type of impact study to conduct depends on considerations such as the budget for the evaluation, the desired level of confidence in the evaluation results, and the practical constraints on conducting an evaluation of a given program. It is important to note that experimental studies (randomized control trials or RCTs) are considered the most rigorous form of evaluation and are often called the gold standard, given that they provide the best scientific evidence of what works or does not. However, they are also the most intrusive type of impact study in that they intervene with program processes. Various types of implementation studies are usually part of impact studies, such as site comparisons and fidelity studies, and of course, outcomes are measured.
4. Cost Studies: A cost study estimates program costs, makes cost comparisons or weighs costs against outcomes or impacts. Cost studies involve analysis of the costs of a program, and some weigh program effectiveness against overall program cost. Sometimes cost documentation, estimation exercises, or simple cost calculations are considered a cost analysis, but not by program evaluators. While they are common elements of all cost studies, they are not considered cost analyses. A cost study draws conclusions about program costs based on systematic cost comparisons (particularly between programs and overtime) or statistical analysis of cost differences or responses to changes in program features or inputs. The specific comparisons and statistical analyses depend on the program and the quality and detail of available cost data.