The Impact Matrix Approach - TI
INTERNATIONAL IMPACT MONITORING APPROACH
Transparency International has long recognised the critical need for better data in the effort to eradicate corruption. Most impact in the anti-corruption field is achieved against a backdrop of complex socio-economic contexts and enabled through dynamic and fast-paced processes involving a range of stakeholders. It can be challenging to measure the impact of our work – the results of our efforts can be unpredictable, and sometimes they take long periods of time to come about. Our progress can also be subject to occasional reversals that need to be understood and explained.
In 2014 Transparency International developed and adopted a new approach to monitoring the impact of anti-corruption work. This approach aims to build a more robust body of evidence regarding what works in the stopping corruption, as well as why and how it works. The Impact Monitoring Approach is designed to be sufficiently flexible in order to capture the different change processes that the Transparency International movement and the broader anti-corruption community are contributing to and achieving.
Background to the Approach
Background to the Approach
- Developed in 2014.
- Impact Matrix: Way of monitoring and analysing our work in order to understand impact and understand how to increase our effectiveness and maximise impact.
- Impact Reviews: In-depth assessments across projects and initiatives to understand at a deeper level how change happens in relation to anti-corruption efforts.
Background to the Approach
This training focuses on the use of the impact matrix for planning and monitoring. IMA is a participatory approach which requires limited technical support (although this is available if needed).
Impact Reviews: These in-depth assessments require significant investment across TI chapters and it is likely that only a few of these will be conducted. The learning from these reviews should be shared across the movement and be integrated within future planning and practice.
A recent example is the assessment of the impact of the National Integrity System Assessment. This used the impact matrix as a lens to understand the types of impact and the internal and external factors that enabled or hindered impact. It also sought to understand how the change came about and what the contribution of particular strategies were to change (what works).
As part of this process 32 national chapters were interviewed. Examples of learning identified were that context is critical to the likelihood of change linked to the NIS – specifically that the NIS has not been effective in authoritarian contexts and that chapters need to have the capacity and commitment to use the findings – for example by having a clear advocacy and communications strategy and internal capacity and external partnerships/relationships to leverage change.
The TI Impact Matrix
The TI Impact Matrix
Focused on Change
Two main areas of change which are mutually reinforcing:
Policy and legislative change: Institutions have all the necessary mechanisms, policies or laws in place to redress and prevent corruption, sanction corrupt behaviour, and promote good governance.
Behaviour change: Individuals, communities, civil society organisations and social movements act systematically to promote global good governance and prevent corruption.
These are supported by:
- Awareness Raising
Policy and Legislative Change
Improved Enforcement of Laws
Law Adoption and Amendment
Better Institutional Processes
Seeking redress against corruption
Monitoring with the Impact Matrix
3 Step Process!
The Impact Matrix Approach is essentially a participatory approach.
1. Map the change and discuss your contribution with you team – this takes place in a workshop
2. Strengthening the evidence – during the workshop you should identify where the gaps in your evidence are and how you can and when you will fill those gaps
3. External Validation – this step may involve a further workshop or stakeholders meeting – the level of investment may depend on your level of confidence in your assessment and the scale and scope of the project. In some instances external validation has been quite small-scale e.g. just having a series of structured individual conversations with key stakeholders (e.g. journalists, MPs).
The IMA is a move away from focusing primarily on indicators. It was designed to address some of the challenges we face in measuring anti-corruption work, i.e. long time-frame, subjectivity, complex contexts, many different stakeholders.
What will you monitor, and when?
Scope and scale
- Thematic area
- Organisational/Chapter level
- Length of project cycle
- When you will see change (incremental or long-term)
- Strategic relevance
Once you’ve set the scope and scale of your impact mapping, you need to organise a kick off meeting and allocate roles and responsibilities. You need to decide who will lead and what resources will be needed. The date of the impact mapping workshop needs to be set and a decision taken as to whether there will be an external validation element of the workshop or not.
1. Writing the story/explaining what happened
This is a summary of what has happened. Where we started, what was achieved, what strategies were deployed and what changed (both unintended or intended). This should be a shared narrative if different stakeholders have been involved in implementation. If you have time it can be helpful to write this story down and share it in advance of any workshop.
In the narrative template – guidance is provided
It is also important to outline what internal and external relationships have contributed to the change and any unexpected changes during the period being reviewed.
- What is the context you are operating in?
- What are the goals and objectives of the projects in the selected strategic areas?
- What is the underlying logic of your projects/ initiatives in the selected strategic areas?
- How is the work expected to benefit people affected by corruption?
- What is your overall sense of success and failure so far for this area?
Map the change
Ideally the monitoring lead should identify concrete examples of impact by going through existing project documents, reports etc. A preliminary mapping of these impact examples against the impact matrix should be completed ahead of the more in-depth discussion that will take place at the mapping meeting.
A meeting should be organised including project team members and key internal stakeholders from other teams (especially if there are shared approaches).
The workshop should include the following elements – we will go through these in greater detail during this workshop.
- Discussion of impact – the preliminary results that have been prepared by the monitoring lead should be used as a basis for discussion – with the team adding other chances and impacts. The discussion should be used to draw out lessons learned and implications for future planning or implementation. It is important to document any unintended outcomes (positive or negative).
Make sure that you have identified someone who is going to document the discussions during the workshop and look for the key elements that need to be recorded in the narrative template.
- Lessons learned
- Unintended outcomes
- Evidence gaps
It may be that the monitoring lead isn’t the right person to facilitate the meeting or to document the discussions (they may want to be too involved in the proceedings) – so think about this in advance.
The team then discusses and rates their contribution against each change that is mapped onto the impact matrix. It is important to also discuss the evidence that backs up the contribution claim (and how it could be strengthened).
It’s important to focus on the change (not what was done). If you find yourself discuss activities – push yourself by asking what the activities were intended to achieve.
Identifying Data Gaps
In order to move on to step 2 it is essential that you are clear what evidence you have to substantiate your claims of contribution and what further information you can gather in order to strengthen the claims and also understand your impact.
It might be useful to rate your confidence of each of the ratings in order to understand where you need to gather more information and evidence.
What information would be useful?
Whose feedback could you seek?
Strengthen the Evidence
During the initial workshop, data gaps were identified and some initial suggestions as to how the missing data could be collected were discussed.
What is good evidence?
and strengthens our confidence that there is a causal link
between our intervention and an observed result.
- It could include feedback from key persons, groups and institutions. We can also draw on any other evidence that supports our claims, such as meeting minutes, reports, media reports, survey results, links and screenshots etc.
- It should be triangulated. Triangulation means using three or more sources or types of data to cross-check reliability and validity.
In step 2, the monitoring lead continues the process to strengthen the impact mapping and evidence for contribution claims
Data gap – suggestion for additional data – method for collection – organise data collection (interviews, focus-group discussion, surveys).
This will take significant time and effort and so the monitoring lead needs sufficient time to strengthen the evidence in advance of any process of external validation.
Validate Your Data with Stakeholders and Partners
Purpose: Validate our impact mapping by getting feedback from our stakeholders (through an external stakeholders meeting or more scaled-back approach)
It is important to note that the external stakeholders meeting can be useful for showcasing the work of the chapter and potentially to engage key stakeholders (funders, parliamentarians, civil society allies). However it is important that the fundamental purpose of the workshop is not overshadowed by the desire to create a good impression on those stakeholders who are attending. It is also important to not just invite stakeholders who agree with you and are likely to share your analysis and arguments. In some contexts, it may not be appropriate to hold a stakeholder workshop (due to lack of civil society space, sensitivity of project objectives and activities) and in this case interviews with individual stakeholders might be a way of managing risks.
- State the purpose of meeting
- Provide an overview of the
project/initiative being discussed
- Present the main changes enabled through the project and who benefited from
these (women, men, minority groups). Is there anything that is not captured by the
impact mapping yet?
- Discuss any unintended or negative consequences.
- Present and discuss the contribution ratings made by the team in Step 1.
- Confirm that our evidence is sound and convincing, and warrants the contribution
ratings. Be sure to accept criticism as a constructive contribution to the validation of our claims.
- Discuss the lessons learned and how to improve future work.
- After the stakeholders meeting, the monitoring lead should finalise the monitoring report begun in the previous two steps.
Quick plenary discussion (or pairs then plenary) – what challenges do you expect in organizing the impact matrix validation workshop? E.g. who to invite, how to approach it with the right attitude
Tips on how to plan a validation meeting, e.g. needs to be adjusted to context and needs. Who has insight into this area of work? Who might have critical views? In some cases an informal coffee meeting might fulfil the purpose, in other cases the validation meeting could be part of a more formal strategic planning workshop with stakeholders.
Analyse the Data
- Assess progress. Changes achieved should be compared to the baseline data and what the project initially set out to achieve.
Ask questions like: Were the main targets hit? Were new aims discovered mid-project? Can we explain why some ambitions were not met?
- Identify impact clusters. The impact matrix can be used as a tool to identify clusters of impact data, highlighting areas of progress and those which were more challenging.
Ask questions like: Were we successful on only one strand of our ambitions, and if so, why?
- Identify trends. Developing trends can be detected, for example, by comparing findings of several projects being monitored.
Ask questions like: Do our projects or initiatives show similar findings/lessons?
- Analyse pathways for change. Examine how certain actions and changes may have contributed to changes and impacts, and what the enabling strategies were that brought about these results.
Ask questions like: How and why did a set of interventions lead to the intended or unintended change? What factors played a role?
- Re-assess assumptions. The validity and relevance of the underlying logic or theory of change should be tested in light of the current context. This can be done by assessing the effectiveness of our approaches in bringing about our intended changes.
Ask questions like: Were the approaches chosen the best in yielding results or could other approaches have yielded similar or better results?
Using the Data
- Produce reports for accountability. The completed impact matrix provides a rich picture of impact across the various components and areas of our work. It helps to demonstrate our work and its impact, and can be used as a basis for an annual accountability report. Robust evidence and a compelling narrative of impact can also be a powerful tool for mobilising additional funds to support our work. Don’t sit on the data – use it!
- Steer strategic direction and allocation of resources. Even if good MEL systems are in place, the impact matrix provides a unique opportunity to develop a coherent and overall picture of our work’s impact. It allows us to identify areas where we are not making an impact, prompts discussions about opportunities for further change, and informs decisions around prioritisation and allocation of resources.
- Identify advocacy opportunities and issues. The impact matrix allows us to discern certain patterns and trends which we can use as compelling evidence for advocacy, as proof of emerging issues worth advocating on, or as grounds to change the direction of our advocacy.
- Strengthen organisational learning. The process of completing the impact matrix with our teams, colleagues and partners provides an important space to periodically reflect on achievements and challenges, learn from them, and draw conclusions for the future. This is particularly relevant in contexts where sophisticated learning systems are not in place. It is important that staff are supported and understand the value of learning about the different areas of their work.