Outcomes measurement software give organisations the opportunity to answer important questions about the effectiveness of their programs by using data to validate program hypotheses and assumptions. The kinds of questions an organisation can answer with their data depends on where they are at in their outcomes measurement journey. In essence, there are 5 questions that an organisation should strive to address. Each question is basically a level of outcomes measurement work and identifies how your organisation is using data to demonstrate impact. These questions include:
- Do we serve a need?
- Are we making a difference anecdotally?
- Are we making a difference empirically?
- How can we improve and grow?
- How can we make a collective impact?
This continuum of questions can be applied to any organisation hoping to achieve more in their outcomes measurement work, with the purpose being to move to higher levels. For example, if you can prove that you are “serving a need,” then the next step is to prove that you are “making a difference anecdotally.” But, you can only move up the continuum if you know where your organisation falls on this continuum and you know what questions to ask to achieve higher levels. This blog will outline each of these levels so you can determine how you may be able to improve your outcomes measurement work.
Level 1: Do we serve a need?
While this is the lowest level of outcomes measurement, it is an important one in that it confirms that needs are actually being addressed. In particular, output data is used to answer this question and defines the following three things:
- Are we serving?
- Who are we serving?
- How are we serving?
So, in order to reach this level, you should do the following:
- Count outputs of the number of people you serve and the number of services or supports you provide.
- Monitor participant demographics to make sure you are addressing the needs of your target population(s).
- Monitor services and supports delivered to make sure you are meeting expectations for the quantity of services and supports offered.
- Understand the needs in your community with outreach or advocacy, and confirm your mission addresses an identified need.
Level 2: Are we making a difference anecdotally?
If you are able to confirm that you are addressing needs, then the next question to ask is “are we making a difference?” However, there are two variations of this question – an anecdotal and an empirical version.
The anecdotal version uses stories, experiences, anecdotes and subjective assessments to outline the difference your programs, services, and supports make on your target population, while the empirical version uses objective data. Obviously it is more challenging to validate these stories empirically, but as a starting point, anecdotal information can be useful for fundraising, development, and communicating your impact externally, as well as building internal buy-in.
To reach this level, you should do the following:
- Track subjective assessments or evaluations of program participants from enrolment to exit.
- Conduct program participant feedback surveys and track results.
- Develop stories of progress made by program participants including “big gains” and transformational shifts.
Level 3: Are we making a difference empirically?
Moving beyond anecdotal evaluations of program impact to empirical evaluations requires a systematic and disciplined approach. If your organisation is at this level, you should be able to answer the question “how do we know we are making a difference?”
Being able to answer the “how do we know” question requires objective proof and validation. Validation comes from tracking, monitoring, analysing, and acting on the right performance data related to your programs and services.
Organisations who are at this level generally have a clear strategic plan clarifying the goals and purpose of the organisation, the reason why it exists and why staff do the work that they do. This is sometimes called a program logic model, which is essentially a strategic framework that defines what the organisation plans to achieve with its programs and services, how it plans to achieve it, and the intended “results” of those plans.
So, to reach this level, you should do the following:
- Develop a methodology that identifies what, how, when, and why for your programs, services, and supports.
- Maintain a data management platform that allows you to objectively collect, analyse, and act on program data and information.
- Build a culture of asking “why,” where curiosity into the “how do we know” question drives your management processes.
Level 4: How can we improve and grow?
Although making a difference is important, it does not stop there. The next level of outcomes measurement is to make a consistent difference by:
- Improving program performance over time through managing, analysing, and acting on data with objective evaluations of program performance to improve outcomes and impact over time; and
- Growing your organisation through expanding its reach to touch more of those in need.
An organisation that is expanding its reach while consistently improving performance can achieve growth in impact.
To reach this level, you should do the following:
- Implement performance improvement methodologies that maintain positive performance trends over time.
- Build a culture that pushes for bigger and bigger results from your programs and services.
- Produce highly effective management systems that set goals, report on progress, and act on analysis.
- Tie program-related goals, strategies, and tactics to larger organisational strategic planning, budgeting, fundraising, and administration (build an ecosystem for organizational growth).
Level 5: How can we make a collective impact?
The four questions above focus on the internal drivers of program performance, or what are known as organisational or program level outcomes. These outcomes refer to the wellbeing of an organisation’s client population and enable organisations to determine if their clients are better off as a result of the work that they are doing.
But these questions fail to address the larger collective impact made by your organisation to actually solve social problems. This is known as population level outcomes or addressing the wellbeing of whole populations or whole communities. These outcomes enable organisations to articulate, through cross community partnerships, the outcomes they would like for a particular population to make progress on the quality of life for that population. So, these outcomes are not the sole responsibility of any one organisation or program, but a collaborative approach by multiple organisations who are all working with the same target group to answer the question “how can we make a collective impact?”.
While this question is an important one, very few organisations are able to answer it. So, if you want to improve your capacities and capabilities in outcomes measurement, you may start out working on program level outcomes, but as your organisation grows, you should eventually work towards achieving population level outcomes in collaboration with other organisations. This level is the pinnacle of outcomes measurement work and should be the goal of every organisation in order to achieve higher levels of impact measurement, performance, and effectiveness.