Impact analysis

Is it enough for a stakeholder to learn that your program produced public value, or do stakeholders want to know how much value was created? Put another way, is it adequate to demonstrate that a program has a positive return on investment for a community? Or does it have to have a higher return than all the alternative investments the community could have made?

I was asked this question today at a Center for Integrative Leadership Research Forum where I presented, “How Cross Sector Leadership Builds Extension’s Public Value.” It seems that the answer has to be yes, it does matter whether a program generates a large or small amount of public benefit relative to its cost.

A potential funder wants to direct dollars toward their highest use. Ideally, all programs would report a return using a common metric. The metric could be dollars for programs whose impacts can be monetized (i.e., converted to dollars); it could be some other common metric (e.g., amount of pollution remediation, high school graduation rate) for programs addressing similar issues. With common metrics, a funder could rank programs by return on investment and fund as many of the top programs as they can.

Such apples-to-apples comparisons must be rare, though, even for programs with common objectives. I also imagine that the magnitude of a programs’ expected public value–if it is known–will inform, but not drive a funder’s decision.

What has your experience been? Have you sought funding from a source that demands to know the expected return on their investment in terms of dollars or some other metric? Do you know of funders that use those metrics to rank programs by return?

Next week I will be in Washington, DC, presenting on Extension’s public value at two venues. If you will be attending either of these meetings, I hope you will share your experiences with documenting public value impacts.

First, on Tuesday, August 6, I will be on a panel at the Agricultural and Applied Economics Association (AAEA) Annual Conference at the Marriott Wardman Park. The session, “Creating and Documenting Extension Programs with Public-Value Level Impacts,” is intended as a resource for AAEA members in their first few years of a partial or full Extension appointment at the university level. In addition to my presentation, “Tell Us About Your Extension Program’s Public-Value Level Impacts,” the panel features Kynda Curtis of Utah State University and Jo Ann Warner of Washington State University Extension. The panel is moderated by my UM colleague, Elton Mykerezi. Ours is concurrent session 2003 from 9:30 to 11:00.

Second, on Wednesday, August 6, I will meet at the Association of Public and Land Grant Universities (APLU) offices with the Measuring Excellence in Extension Implementation Team. I’ll be presenting on how to effectively incorporate public value into impact statements.

At both of these meetings, I’ll be eager to hear about public value work and impact analysis taking place across the country.

My UM Extension colleague Neil Linscheid has alerted me to this recording of Harvard Kennedy School’s Mark Moore discussing his new book, Recognizing Public Value. I haven’t read the book yet, but I’ve ordered it from the library, and I’ll write more once I’ve read it. However, I can share some quick impressions from having listened to the recording of Moore’s talk.

Both Moore and his discussant, Tiziana Dearing of the Boston College School of Social Work, commented on the limitations of monetizing public value created by service providers. Starting around 28:00 of the recording, Moore explains that to measure the value created by government, we are often challenged with “describing material conditions that are hard to monetize….We don’t know how to convert them into some kind of cash value.” He doesn’t give examples, but it’s easy to imagine changes in civic or social conditions as falling into this category. Moore then decries the amount of effort that goes into trying to monetize these effects and says that effort “mostly distorts the information rather than clarifies.” For clarity, he would prefer a concrete description of the effect (presumably something like units of pollutant reduction or increase in graduation rates) to “somebody’s elaborate method to try monetize it.”

Dearing shares Moore’s skepticism of monetizing government and nonprofit benefits, and around 42:00 of the recording says, “It’s a very dangerous thing outside of the private sector to have the same enthusiasm for the bottom line.” Later she warns that “it’s so easy to follow the metrics that follow the dollar that it becomes shorthand for a theory of change.”

In this post, I also urged caution about monetizing the impacts of Extension programs. Nevertheless, when they are carefully framed and rigorously executed, dollar-denominated metrics such as cost-benefit analyses allow for comparison across programs. I wonder how Moore and Dearing would advise a local government that is choosing between anti-pollution and education investments when the only impact data they have is a “concrete description of the effects”? How would they balance units of pollutant mitigation with a percentage change in the graduation rate?

Have you read Moore’s new book or at least listened to his talk? What do you think about what Dearing calls our “enthusiasm for the bottom line”? Do we go too far in trying to translate program impacts into dollars? What other contributions of the new book are particularly relevant for Extension?

Nancy Franz of Iowa State University Extension has alerted me (and the rest of the her Public Value Network listserv) to the California 4-H program’s involvement of volunteers and staff in a statewide effort to develop public value stories. On the program’s California Public Values web page, a survey link invites volunteers and staff to participate in the effort to “shape California 4-H’s public values that will be shared with the broader community that has a stake in 4-H.”

The survey asks respondents to identify (1) positive benefits to youth from participating in 4-H, (2) ways society benefits from those positive youth outcomes, and (3) community or state benefits from adult volunteer participation in the 4-H program.

“Crowdsourcing”–at least from select sources, such as those working with 4H across the state–sounds like a potentially efficient way to assemble a body of program impact stories. Local program staff can bring to administrators’ and evaluators’ attention impacts that might otherwise have been overlooked. I am eager to hear how the California 4H public value project develops.

At last week’s 2013 PILD conference, I heard about a couple of national initiatives that, once developed, should help Extension organizations share impact data that can inform public value messages. In my own comments at the conference, I supported cross-state sharing of ideas, so I was encouraged to hear about these national projects.

First, I think it was NIFA director Dr. Sonny Ramaswamy who mentioned an effort this summer to develop a national portal for impact reporting that will consolidate Extension program impact results.

Second, I believe it was ECOP chair Dr. Daryl Buchholz who highlighted ECOP’s Measuring Excellence initiative. It appears that this project is meant to define and demonstrate excellence and to report impacts for Cooperative Extension as a whole. From the website: “Cooperative Extension has advanced from merely reporting inputs and outputs to documenting outcomes and impacts of its programs. However, most of these measures are tied to specific programs. They are not generally assessed or considered at the organizational level.” While the “excellence” part of the website is well-developed, the pages having to do with impact are still under construction. I look forward to the work that will populate these pages with resources and guidance for Extension impact teams. Meanwhile, I have to give a shout out for the public value statements on the front page of the website!

By the way, my notes from the PILD keynote talks are a little sparse. If I am wrong about which speaker spoke about which initiative, please correct me in the comments. And if you know more about how the portal or the Measuring Excellence project can strengthen Extension’s public value case, please share that, too!

(Photo credit: USDAgov on FlickR)

Earlier this week I spoke at the 2013 PILD Conference in Alexandria, VA. The title of my talk was “You can understand and effectively communicate the public value of Cooperative Extension,” but once I was done preparing my remarks, the title might as well have been, “In search of the elusive public value-level impacts.”

Because the goal of the public value approach is to persuade stakeholders–specifically, the non-participant payers of Extension programs–we must credibly argue that our programs will lead to impacts that affect the whole community, not only the participants. The Holy Grail of the public value approach, then is public value-level impacts. In the parlance of the UWextension Logic Model, these are long-term outcomes or impacts: Changes in social, economic, environmental, or civic conditions. Improved water quality, reduced public health costs, greater economic vitality and resilience, higher property values, lower taxes, lower crime rates, more effective leadership.

To address this challenge, Extension organizations can begin with the end in mind, where the “end” is those elusive public value-level impacts. The fact is, not all of our programs were designed to achieve condition changes, and few come packaged with evaluation plans designed to document that level of impact. Keeping that need in mind can help us address this deficit as we move forward.

The action steps in the previous slide illustrate the inherent multi-disciplinarity of the public value approach. For each of the action steps, I have listed the skills or disciplines that I thought could contribute to the success of that step. For example, to effectively determine which public value-level impacts to pursue will require experts in community engagement, facilitation, public and legislative relations, and grant-writing as well as the program area’s subject matter scholars. You may disagree with my list or think of others to add. But we can probably agree that if multi-disciplinarity is a key ingredient to developing strong public value messages, then Extension, as a naturally multi-disciplinary system, is well-positioned to create its own future.

Another encouraging observation is that no Extension organization has to tackle the public value challenge alone–nor should it. We are all working on the same action steps for programs that have similarities across states. Working together and sharing ideas across states we can learn the best ways to tackle each of the steps, and make more efficient use of our resources.

Finally, putting these ideas together can help us strengthen Extension’s public value case and contribute to the organization’s sustainability.

Were you at the PILD conference? Do you agree or disagree with my comments? What did I miss that is important to your organization’s ability to make a public value case?

Today in her blog, “Food Politics,” Marion Nestle summarizes some recent estimates from the Campaign to End Obesity of the cost of obesity in the United States. Nestle sounds reluctant to take the estimates at face value, but admits, “One thing is clear: obesity is expensive, personally, economically, and politically.”

Extension programs of many types aim to reduce the incidence of and costs associated with obesity. Not only nutrition education programs, but some agriculture programs (farmers markets, local foods), youth programs, and even community leadership programs seek to improve health and impact the obesity problem. And to be effective with non-participant stakeholders–to demonstrate public value–these programs would like to report the dollar value of savings they have helped generate for state and local governments and for communities. Can estimates such those released by the Campaign to End Obesity help Extension programs make their case?

Do you work with programs that address obesity and its impact on communities? Have you found estimates of the cost of obesity that can be used to support your program’s public value? Do you focus on public health costs–which affect all community members–or have you found a way to incorporate even the impact on private health care costs into a public value message?

In public value workshops, we emphasize that Extension programs’ impacts create value for people who have no contact with the programs. We argue that creating value for these stakeholders–the greater community–helps us make the case for public sector funding for Extension programs.

Today at the 2012 Virginia Cooperative Extension Professional Development Conference, I heard Jill Bramble, Chief Development Officer for the National 4-H Council, make a case for private sector investments in Extension programs. Jill suggested that Extension organizations view private businesses as potential investors who have a stake in the impacts that our programs generate. When a local 4H program increases the supply of prepared workers in a community, local employers benefit. When a nutrition education program increases demand for farmer’s market produce, local farmers benefit. By highlighting these kinds of impacts, we can help private businesses see how they will reap rewards by investing in Extension programs. As Jill put it, our programs generate the impacts, and businesses “want to buy that impact.” Jill said that businesses are eager to engage in “impact investing,” but we need to know and tell our programs’ impact story.

Whether we want to make a case for public investments or private investments in Extension programs, it seems to all come down to measuring impacts.

Have you generated private funding for Extension programs? Were program impacts an important part of the case for that funding? What approaches have worked for you?