Several factors apply to the decision-making processes for research design and data collection approaches, such as the following:
Methods to Most Accurately Answer Key Research Questions: Some questions, such as who is participating in a program and the characteristics of their participation, may be best answered with an implementation or descriptive outcomes study, whereas other questions about the effectiveness of the program are likely to be best answered with a pre-post outcomes study, RCT, or quasi-experimental study. You may want to conduct a study that includes several types of evaluation. For example, often an outcome or impact study will also include implementation and/or cost study components. The key guiding factor in making the final choice of study design is what the agency wants to learn and why, and how certain it wants to be about the findings.
Organizational Capacity to Participate in the Evaluation: Consider how the evaluation activities will blend into the implementation activities of the program, system, strategy, service, activity, or intervention included in the study. Discuss the feasibility and options to carry out and participate in the selected evaluation with organization or agency managers or operators that implement the subject or topic of the study. Include other key stakeholders and partners to identify their organizational capacity to participate. For example, RCT evaluations of service delivery interventions integrate a random assignment process into the participant intake and enrollment processes that may span multiple partners or service providers. Each partner engages in discussions and negotiates agreements to participate in the evaluation.
Organizational Capacity to Conduct the Evaluation: The selection process for an evaluator also depends on the agency’s capacity to conduct the evaluation. While funding may be a driver to building evaluation capacity, investments in evaluation management development, staff training, strategic and long-range planning, budgeting, and technical assistance are key elements. Each of these elements supports organizational evaluation capacity, whether the evaluators conduct in-house studies, partnerships are formed with research universities to manage administrative data, or third-party evaluators are procured to conduct independent evaluations.
Data Availability for the Desired Type of Evaluation or Capacity to Collect It: In addition to a final determination of evaluation type(s) for the evaluation plan, the evaluation planner and the selected evaluator will finalize the details to conduct and carry out the research— research design (i.e., methods) and data collection approach. The type of evaluation and the key research questions will present different methods, data sources, and data collection options. Data availability or the agency’s capacity to collect data are critical factors in deciding the type of evaluation to conduct. The selected evaluator will refine the study methodology and data collection approach; however, as part of the state evaluation planning process, data availability and capacity to house, transmit, and secure the data must be addressed to put the evaluation on the right track. How the data needs are identified and resolved within the context of the evaluation methodology allows the evaluation planner to work most effectively with the selected evaluator— in-house unit, partner university/organization, or third-party entity. See page 42 of the DOL Framework to see a table that explains the key elements of the evaluation plan.