What Do We Mean by “Evaluation”?
A network of primary health care practices in New York State secured grant funding to start a new program to improve care for its patients with diabetes. The funding was for an education program for health care providers about best practices for diabetes care, with an emphasis on the need for regular patient visits and blood sugar testing. Early on, the program manager had several questions about how the project was going, including what worked and what did not. She was also interested in gathering information that could showcase her program’s impact to the community and to her funder. To get this valuable information, the program manager planned a program evaluation.
What is evaluation?
Evaluation is a way of assessing the effectiveness of a project or program and of measuring whether its proposed goals have been met. Program managers and staff often use evaluations to improve program effectiveness. Program evaluation and related activities can answer questions on whether a program is working or how it can be improved.
What kinds of questions can an evaluation ask?
An evaluation can be guided by several types of questions:
1) Implementation: Are the program’s activities put into place as originally intended? What adaptations were made and why?
2) Effectiveness: Is the program achieving the goals and objectives it was intended to accomplish?
3) Efficiency: Are the program’s activities being produced with appropriate use of resources, such as budget and staff time?
4) Cost-Effectiveness: Does the value or benefit of achieving the program’s goals and objectives exceed the cost of producing them?
5) Attribution: Is the progress on goals and objectives a result of the program, rather than to other things that are going on at the same time?
What are the benefits of conducting an evaluation?
There are many benefits to conducting an evaluation. First, designing an evaluation opens up communication among the leaders of an organization, the managers, and the staff. Doing so can encourage analytical thinking and allow for honest discussions about the program among everyone involved with the project. In addition, conducting an evaluation provides an opportunity to revisit the goals of an existing program, and to bridge any gaps that may exist between the vision of the program and the reality of the program operation.
What are the challenges of conducting an evaluation?
Conducting an evaluation is not without its challenges: operating with limited resources (e.g., time, money, and/or expertise) and limited experience with the evaluation process (e.g., a lack of understanding about the difference between terms like outputs and outcomes). The goal of this NYSHealth evaluation planning guide is to provide information and resources that minimize some of these challenges.
Topics covered on this site (click on topic for more information):
Policy or Advocacy Activities: Policy and advocacy projects focus on activities to influence health care policymakers, including elected officials, health systems, payers, and providers. Activities may include public education, capacity building, network formation, relationship building, communications, and leadership development.
Logic Models: A logic model is a visual representation of a program. The logic model can help define a program, create a shared understanding of what the program is, and provide a starting place when planning an evaluation.
Process Evaluation: Process evaluation assesses how a program is implemented. Also called “formative” evaluation, process evaluation focuses on the process, rather than on outcomes.
Outcome Evaluation: Outcome evaluation measures the actual changes resulting from program activities (i.e., the outcomes) and assesses the program’s effectiveness.
Measures: Measures (or indicators) are the observable and specific items of information that will be collected during the evaluation.
Research Design and Sampling: A research design is a plan outlining how information will be gathered for an evaluation. Typical tasks include identifying how data will be collected and by whom (e.g., the methods), determining the data collection materials to be used, and how the information will be organized and analyzed. Sampling describes who will be studied or included in the evaluation (e.g., providers, patients).
Qualitative Research: Qualitative research uses non-numerical data (from focus groups, interviews, etc.) to tell the stories behind the numbers. Qualitative data results in rich, contextualized descriptions of abstract information.
Quantitative Research: Quantitative research uses numerical data (from surveys, administrative data, etc.) that can be analyzed using statistics. Findings can be generalized to the population that is being examined.
Internal versus External Evaluation: There are many considerations to take into account when deciding whether to conduct an in-house evaluation or to seek out outside help. Some factors to take into account are the need for independence and objectivity, need for technical expertise, and time and resource constraints. Read more for guidance about whether to hire outside expertise.
Resources for Further Information: A reference list of further resources on a variety of program evaluation topics.