This article was originally posted on Benckiser Stiftung Zukunft.
Everyone, including us, is talking about impact. Some talk about impact so much that their audience rolls their eyes as soon as they hear the “I”-word. In Germany at least. It’s very well possible that people feel overwhelmed and unsure of exactly how to deal with this new impact phenomenon. We believe this reaction is largely due to the still unclear, indiscriminate, and inconsistent use of the term “impact”.
What is impact? First of all, causality is inextricably linked with impact. So it’s impossible to have a conversation about outcomes and impact without discussing cause and effect, the two sides of causality. In the social sector, the terms outcomes and impact, describe when outreach activities successfully enact medium- and long-term changes within a specific target group, to their environment, and, finally, to sections of society. Therefore, there is a causal relationship between social activities and these changes.(1)
For a long time, no one paid attention to if or how social programs actually worked. The assumed impact and, last but not least, the goodwill behind the program were considered sufficient. Now we know better: not only can social programs not work as intended, but they can also be completely ineffective, miss the target group, or even be harmful. It should be self-evident that the players in the social sector do everything in their power to reach as many people as possible in their target group, as successfully as possible, and that this success can be proven to the best possible extent. Even more so, since most of this work is financed by taxpayers' money. The current fixation on impact should therefore be a welcome change. However, only if the term and concept are used consistently and appropriately.
What is necessary when talking about a program’s impact? First, you should know what you really hope to achieve with a program, that is, have concrete and detailed project goals. And then, check if and how exactly the respective program achieves these goals.
How can I check for impact? It seems that many foundations, non-profit organizations and consultants have difficulties with this. It is actually quite simple: an examination of cause and effect can lead the way. As long as I cannot show that a demonstrable change in the target group happened as a direct result of my social activity, I cannot and must not speak of impact. However, scientifically valid evidence of a causal relationship between intervention and change is not always easy to provide. That is to say, the new focus on and demand for impact is not without significant barriers. Impact is, after all, no small thing. This also explains why, of the too-few organizations that want to prove their effectiveness, fewer still can actually provide this proof.
How do I avoid blowing out the delicate Impact-Flame before it has a chance to ignite into something greater? The State Prevention Council of Lower Saxony (Landespräventionsrat Niedersachsen) is working remarkably thoroughly on the topic of effectiveness in its project "Communities That Care - CTC". Their recommendation list "Grüne Liste Prävention" (Green List of Prevention Programs) only includes programs with preventative approaches that meet a set of clearly defined requirements. The basis for these requirements is the theoretical model from Jan Veerman and Tom van Yperen(2). In this model, social programs are assumed to be developing on a progressive scale where descriptions, theoretical explanations, or proof of target achievement through indicators can be applied. But these can only be precursors to the actual proof of a causal relationship. "The lower the level of a program's development , the more evaluation concepts are suitable. For proof of efficacy on the causal level, only the evaluation methods which also allow causal statements are possible."(3)
Therefore, the listed programs are sorted according to five criteria – the program’s concept and implementation quality, and additionally the applied evaluation’s standards, the validity of the results and the results themselves – into three categories:
- Effectiveness is theoretically well-founded
- Effectiveness is probable
- Effectiveness is proven
Overall, the current impact debate would benefit highly from such a differentiated approach. It should be the standard for those who are currently talking about impact. As clear as it is that impact can only be claimed where it has been causally proven, it is also important to recognize that many organizations will likely never be in a position to provide such strong evidence of their impact, for example, because they are too small.
With that in mind, donors should stop demanding evidence, or reporting proven impact, where there is none now or none likely in the near future. Instead, they should support organizations in their development, which can in turn allow these organizations to "formulate their hypotheses about causal relationships more explicitly so that they can (...) be part of tests [with causal evidence]"(3). Or, they should simply accept that they support, in good faith, an organization that is structurally unable to prove its impact.
If the goal is to provide proof of impact in the framework of a scientifically-recognized research design, we consider the term "impact measurement" to be useful. Anything that does not stand up to scientific scrutiny, because there is no proof of causality, must be named differently. Here, the term "impact orientation" may be of help - as a description of the stance of social organizations that have oriented themselves toward defined goals which could serve as the basis for an impact assessment. Still, alone, neither this impact-oriented attitude, nor impact-oriented processes or structures, can justify the statement: “The program achieves impact!” No one would claim that a person who would like to save energy at some point, is already actually contributing to environmental protection. The same is true for impact.
But even if the impact measurement is successful and a program can prove that it actually achieves the desired positive effects, there is yet another crucial question to be answered: Is the impact achieved at a reasonable cost? After all, the question of how to reach as many people as successfully as possible, given the limited resources available, is always connected to the question of whether we can achieve this impact with as little money as possible.
Only when both questions have been answered, the question of impact and the question of the costs per impact, are public and private donors able to make decisions based on sound evidence rather than well-intentioned dreams. Then we could have a world in which the only one who is allowed to speak about impact is the one who can prove it and therefore then be the one who receives the support that their work deserves.
References:
(1) see (within the German context): Phineo gAG (ed.) (2015): Kursbuch Wirkung. 3. Auflage, p. 5 (https://www.phineo.org/fuer-organisationen/kursbuch-wirkung); Social Reporting Initiative e.V. (ed.), Präsentation „SRS Workshop“, slide 10 (http://www.social-reporting-standard.de/srs-materialien/srs-foliensatz/)
(2) Veermann, J.W., van Yperen, T.A. (2007): Degrees of freedom and degrees of certainty: A developmental model for the establishment of evidence-based youth care, Evaluation and Program Planning 30, p. 212 – 221
(3) Landespräventionsrat Niedersachen (ed.) (2011): Grüne Liste Prävention. Auswahl- und Bewertungskriterien für die CTC Programm-Datenbank, p. 8 (http://www.gruene-liste-praevention.de/communities-that-care/Media/Grne_Liste_Bewertungskriterien.pdf)
This article was originally posted on Benckiser Stiftung Zukunft.You can find more articles like this on their Z Blog.
Originally published November 15, 2017
