Program evaluation

The authors conclude that “more higher-level evidence of impact is needed.” They write that Extension’s “ultimate goal is to remain relevant and of value to the public. The strongest method to demonstrate relevancy and public value is to document “true impact” (end results/long-term outcomes).”

Do the authors’ findings ring true for you? Are only a small percentage of programs able to demonstrate public value-level impacts? Is it because few programs are achieving that level of impact? Or because the resources have not been available to measure programs’ long-term impacts?

There has been a lively discussion about communicating the public value of youth development programs on “Youth Development Insight,” a blog of the University of Minnesota Extension Center for Youth Development. Community program specialist, Joanna Tzenis argues that youth programs can create public value by having society-level impacts. Examples of two such impacts are building trust among community members and youth becoming agents for change in their communities. You can check out the discussion here.

University of Missouri Extension’s Stay Strong, Stay Healthy program (SSSH) is a strength training program that leads older adults to feel more active, flexible, and energetic. The SSSH team has created a public value message that they use to generate awareness and support for the program. Here is the message, complete with an estimate of health-related cost savings, excerpted from the program website:

“When you support MU Extension’s Stay Strong, Stay Healthy program, participants will increase their physical activity and may improve strength, balance and flexibility, resulting in reduced risk for falls, better overall health and greater independence. These health benefits decrease the likelihood of a participant entering a nursing home, which costs on average $24,455 per year in Missouri. The money saved benefits the community by keeping more discretionary income in circulation locally. It also keeps people actively, independently contributing to society longer.”

Do you have a similar program in your state? How do you explain your program’s public value?

Looking for ways to support your public value message? Spend some time exploring the Evidence Based Living Blog, written by Cornell Cooperative Extension’s Karl Pillemer, Associate Dean for Extension and Outreach, and Rhoda Meador, Associate Director of Outreach and Extension in the College of Human Ecology as well as the Associate Director of the Bronfenbrenner Life Course Center.

Consider adding the Evidence Based Living Blog to your blog reader, so you can see when the authors post about the latest research or media stories on youth behavior and health. Take a stroll through the archives and read the discussions and evidence assembled therein. You may come across ideas for new research projects or findings that you can use to make the case for your own programs. If you find something that you find useful, go ahead and share that in a comment on the Cornell blog or here. .

Regarding the evaluation methodology for the venison program, Jonathan reports: “For years, we only asked questions like ‘did you pick up some butchering tips,’ or ‘did you learn something about food safety,’ etc. Last year, however, we decided that since we have many return attendees, we would begin asking them if they 1) hunted or fished more as a result of attending our program (we also do fish programs), and 2) do they keep or bring home more fish and game as a result of our programs.”

With affirmative responses to those evaluation questions, Jonathan and his colleagues argue that the hunter/fisherman programs create public value by generating hunting and fishing license fees for the state (provided that the program participants hunt and fish in concordance with state regulations). Moreover, wild game and fish are low in saturated fats and sodium, and are generally part of a healthy diet. Sportsmen and women who bring home more wild game and fish and incorporate it into their diets may see improvements in health. When these health improvements lead to lower public health costs, we can see that the Extension programs have generated public value.

Additionally, if the venison team can produce evidence that program participants identify and report animals that show signs of chronic wasting disease, they may be able to make a “natural resource protection” argument, as well.

Do you have hunting and fishing programs in your state? Have you tried to make a case for public funding for such programs? How do you explain the programs’ public value?

On May 18 in Madison, WI, I will lead a breakout session at the National Extension and Research Administrative Officers’ Conference (NERAOC). I will present an overview–and the basic concepts–of the “Building Extension’s Public Value” workshop, and talk about how to make a case for funding for outreach, extension, and research. If you are planning to attend the conference, please join me at the 10:15 session.

What should an Extension program team have on hand to draft a public value message that secures a skeptical stakeholder’s support? Here’s my prescription:

What’s yours? 

Substantiating the claims that we make about Extension programs’ public value is crucial to Extension’s credibility. However, we don’t always have enough time in a “Building Extension’s Public Value” workshop to assemble the documentation (journal articles, program evaluation reports, etc.) to support the claims embedded in a newly drafted public value message. The purpose of the “Research Agenda” workshop module is to list those claims and create a plan for assembling the supporting documents, or even for conducting new program evaluations or research.

Cynthia Crawford, Family Financial Education Specialist and County Program Director for University of Missouri Extension in Saline County, MO, has a suggestion for helping workshop groups stay creative while not veering too far off into “aspirational” territory. Cynthia suggests telling teams drafting public value statements that they don’t have to have the documentation to substantiate their claims today (during the workshop), but they do have to believe the statements are true today. Cynthia reports that this bit of direction has lead to remarkably strong–and credible–draft public value messages in short amounts of work time.

I will definitely adopt Cynthia’s “you have to believe it today” guidance the next time I teach a BEPV workshop. Do you have any other suggestions for helping teams “think big” while staying grounded?

While the objective of the “Building Extension’s Public Value” workshop is to draft a qualitative message about a program’s public value, many of our stakeholders are concerned about programs’ financial impacts. For example, county commissioners and state legislators want to know how much a program will cost, and whether it’s impacts will reduce strain on the county or state budget. A lot of us, therefore, are eager to quantify the impacts of Extension programs and, wherever possible, convert those impacts into dollars and cents.

Some exciting work is being done on monetizing Extension program impacts. These economic impact briefs from Texas AgriLife Extension are a strong example, and I know there are many more studies.

In future blog entries, I’ll write more about ways researchers and program evaluators are quantifying and monetizing Extension program impacts. However, as persuasive as a dollars-and-cents case can be with some stakeholders, I can think of two reasons to proceed with caution as we pursue more financial and fiscal impact studies.

First, Cooperative Extension does not yet have all the resources and tools necessary to estimate the financial and fiscal benefits of all of our programs. To do a credible job, applied economists, program evaluators and others would need to devote many more hours to this effort than are currently available. Data must be collected and analyzed, models built and tested, reports written and vetted. The likely result of pressuring program teams to estimate financial impacts while providing them with inadequate resources is a collection of poor quality analyses that erode Extension’s credibility.

Second, some programs’ public value lends itself more readily to monetization than others. For example, a program that helps reduce a county’s cost of managing its waste can make a strong, straightforward, dollars-and-cents case. On the other hand, methodologies for estimating the fiscal impact of social capital improvements are less well-developed.

Because so many of Extension’s stakeholders are concerned about monetary value, I am concerned that those programs whose public value is more easily monetized will rise to the top of the priority list–not because they contribute more public value, but because their value is easier to translate into currency.

The objective of the BEPV workshop is to make strong qualitative cases for all Extension programs that create public value. I hope we can keep doing this, even while we seek the resources necessary to estimate the financial and fiscal impacts of those programs for which that is possible.  

Many of you use the University of Wisconsin Extension logic model to guide program development and evaluation. Below is my first attempt at mapping the elements of the logic model to a public value message.

The “short-term” or “learning outcomes” in the logic model are a means to achieving the behavior changes and outcomes contained in the public value message. These learning outcomes lead the way to public value–and we must identify and measure them–but they are not the focus of the public value message. A skeptical stakeholder is unlikely to be persuaded of a program’s value be hearing that a participant learned or became aware of something. The stakeholder is concerned with what the participant actually did with that knowledge.

What I call “changes” in the public value message are called “intermediate” or “medium term outcomes” in the logic model. What I call “outcomes” are the logic model’s “long-term outcomes” or changes in conditions.

It seems to me that public value typically arises from a program’s long-term outcomes. In some cases, a program’s logic model will already include the outcomes that a stakeholder cares about (public value). In other cases, the public value exercise will tell us which additional outcomes we need to monitor–how we should extend the logic model–in order to substantiate our public value messages.

I believe that the public value approach must work hand in hand with program evaluation: it is through good program evaluation that we are able to make credible statements about our programs’ public value.