Key Questions to Inform Your Evaluation Plan
Below are some questions to consider when writing the evaluation section of your NYSHealth grant proposal. It is not necessary to address each of these questions directly in your proposal—these are simply suggestions to help you frame your thinking and guide an evaluation plan that makes sense for your proposed project.
1. What kind of project have you developed? The type of project you propose can help determine which kind of program evaluation is most appropriate. NYSHealth supports the following types of projects:
Policy/Advocacy-Oriented Projects: Policy/advocacy projects focus on activities to influence health care policymakers, including elected officials, health systems, payers, and providers. Read more on evaluating these types of activities.
Direct Service-Oriented Projects: Direct-service projects typically aim to address identified needs of a community, organization, or population through specific activities, services, or actions. For more on evaluating these types of activities, continue to Question 2.
2. Will you do a process evaluation, an outcome evaluation, or both? Decide which type of program evaluation best fits your project or the specific results you hope to achieve.
Process Evaluation assesses how the program is implemented, and focuses on program resources, activities, and outputs (the direct products of program activities). Process evaluation allows applicants to take the pulse of a program’s implementation, and can answer questions about program operations and service delivery.
Outcome Evaluation measures the program’s outcomes and assesses program effectiveness. Outcomes are the actual changes resulting from program activities, and can include short-term outcomes and long-term outcomes. It is important to ensure that the outcomes measured are realistic and reasonable within the duration and scope of the evaluation—often, for grants of one or two years, evaluations focus on short-term rather than long-term outcomes.
It is often useful to examine both process and outcomes when evaluating a program. Combining the two demonstrates both if a program was implemented as intended and whether it had the intended effects.
Imagine that a health care agency decides to conduct a process evaluation of an education program for health care providers on best practices for diabetes care. Questions that could guide the process evaluation include:
- What are the components of the program?
- How satisfied were health care providers with the program?
- What motivated providers to enroll in the program?
- What (if any) barriers were there to providers’ participation in the program?
- What is the average cost per provider participant?
If the same agency decides to conduct an outcome evaluation, questions would focus on the actual changes resulting from the program activities and might ask:
- Did provider knowledge about best practices increase because of the program?
- Did providers who participated improve their monitoring of patients with diabetes?
- Was there an improvement in patient care (e.g., increase in the number of appointments, number of nutrition consultations ordered.)?
- Did the program have any unexpected effects?
3. What will your evaluation measure?
Measures (or indicators) are the information that will be collected and/or analyzed during the evaluation. Measures can be drawn from existing data or may need to be collected specifically for the evaluation.
Guided by the process evaluation questions listed above in Question 2, the health care agency might develop the following measures:
- Number of providers who attended the program.
- Number of providers who completed the program (e.g., CME credit awarded).
- Number of training sessions that providers received.
- Level of providers’ satisfaction with program.
- Cost per provider participant.
And it might develop the following outcome evaluation measures:
- Number and percentage of providers with an increased understanding of best practices, or average amount of improvement in knowledge.
- Number and percentage of providers who increased the number of nutrition consultations ordered for their patients, or average increase in the number of consultations ordered.
- Number and percentage of patients with diabetes who attended at least one nutrition consultation during the year.
4. What is your research design and sampling plan?
The Research Design is the glue that holds a project together—it is a plan for outlining how information is to be gathered for an evaluation. Research designs can be complicated or simple, and describing all the options is beyond the scope of this guide. The most appropriate design will depend on an evaluation’s purpose, need for scientific rigor, resources, and more.
Imagine that the health care agency wants to explore whether its education program (the intervention) results in providers ordering nutrition consultations for more patients with diabetes. For providers who complete the program, the evaluation might compare the percentage of nutrition consultations ordered for patients with diabetes during the year before the program and for the year after. This is a simple pre- and post-intervention design.
This design can be strengthened by the addition of a comparison group—the agency might also collect data on the percentage of patients for whom a nutrition consultation was ordered by a group of providers who never attended the program. Including a comparison group will give added confidence that any differences seen are, in fact, a result of the program and not just a random occurrence. Not all evaluations will have the need or resources for a comparison group in their evaluation. For more information on constructing a comparison group, see Research Design.
The research design should also outline a Sampling plan, which describes who will be included in the evaluation. Sometimes, an evaluation will use data from the entire population. Other times, this is not necessary or feasible, especially if the evaluation involves collecting new data. For example, if the health care agency wanted to learn about what motivated providers to enroll in their education program, they might carry out in-depth interviews with the participants. But in-depth interviews take a lot of time and resources, and if the program is large, it is probably not necessary to interview every participant. Instead, the evaluation might choose a random sample of 20 provider participants to interview.
5. What data will you use to evaluate your program?
There are often many sources of data that can be used when carrying out an evaluation, including financial records, program-monitoring documents, internal reports, administrative records, and client databases. Primary data, or data collected specifically for the purpose of the evaluation, can include surveys, interviews, or observations. Secondary data are any data that have been collected for another purpose, but might be useful for the evaluation. This might include patient chart reviews, minutes from board meetings, or local census data.
To collect data for a process evaluation of the provider education program, the health care agency might use the following data sources:
| Measure | Data Source |
| Number of providers who attended the program | Attendance records |
| Number of training sessions administrated | Training materials |
| Level of satisfaction among providers | Training evaluation forms |
To collect data for an outcome evaluation, the health care agency might use the following data sources:
| Measure | Data Source |
| Number and percentage of providers with an increased understanding of best practices | Pre- and post- intervention surveys (written) |
| Number and percentage of patients with diabetes who attended at least one nutrition consultation during the year | Chart review |
6. How will you analyze your evaluation data and information?
Qualitative Research uses non-numerical data to provide rich, nuanced narratives about experiences and behaviors. It is the stories behind the numbers. Qualitative data (e.g., focus groups, interviews, patient notes) are analyzed to find themes or patterns that help to explain key relationships.
Quantitative Research yields numerical data (e.g., Body Mass Index, cholesterol levels, blood pressure measurements), usually analyzed through statistics. Descriptive statistics do just that—describe the data, and include totals, percentages, and averages. Inferential statistics help to determine whether differences are statistically significant (i.e., whether differences are unlikely to have occurred by chance). Inferential statistics can be used to draw conclusions beyond the sample group to the larger population.
Evaluations can generate a great deal of data and information. How that data and information are analyzed should be driven by the evaluation questions and the type and amount of data collected. The approach to data analysis can affect a research design and sampling plan, so it is important to develop the data analysis plan upfront.
7. Will you carry out an internal evaluation or hire an external evaluator (or some combination of the two)?
There are many considerations to take into account when deciding whether to conduct an in-house evaluation or to seek out outside help. If an evaluation requires independence and objectivity or is highly technical, or if staff time and resources are limited, an external evaluator might be the better choice. Alternatively, if a program has little money to spend on evaluation and has internal expertise, an internal evaluation might be preferable. Hybrid models (internal and external team members) can combine the best of both worlds.
Read more on Internal versus External Evaluation for guidance on the decision whether to hire outside expertise. For help locating an external evaluator, see Finding an External Evaluator.