Publish What You Fund’s 2016 Aid Transparency Index reported on the results by major donors to meet the commitments made in Busan in 2011, where donors promised to open up their aid information by December 2015.

With this deadline now passed, our intention is to raise the bar on aid transparency, beginning with a consultative review on the Index methodology. Over the past few months, we have received over 60 submissions from governments, civil society organisations and researchers from around the world in response to our Guidance document. We are grateful to everyone who spoke with us and provided feedback. The open consultation has now closed.

This process has enabled us to finalise some decisions on the updated methodology, and there are other issues which require further work. We will take the next step in this consultation process by reaching out directly to Index peer reviewers, donor agencies and experts, as well as attendees at the IATI Technical Advisory Group meeting (TAG) in Tanzania in March. We also hope to engage with data users and publishers there. This will be followed by a technical consultation on the Github platform, which is open to all interested parties.

Agreed changes

We proposed a number of changes in the methodology consultation. They covered five areas: the components of aid transparency; the indicators used for assessment; the weighting attached to specific indicators; the scoring system; and a number of additional areas including data use and sampling.

There were many areas where we saw consensus or near-consensus for these from the submissions, and we will modify the methodology accordingly. They are:

• Components: Most respondents agreed that the six components and four principles that we identified in the Guidance document are a useful and accessible way of describing aid transparency. We heard that the term ‘local level monitoring’ could be confusing, so we are working on a new title for that component and welcome additional suggestions.
• Indicators: Most respondents agreed that the list of indicators we proposed was relevant and comprehensive. We will implement the changes proposed in the Guidance document. However, some people suggested we should retain the ‘description’ indicator as several organisations find this information useful for their efforts to improve transparency. As a result, we have decided to retain that indicator and drive better quality by updating the definition of and test for the indicator. Further details on this will be provided as part of the Github consultation.
• Weighting: We will implement the changes in weighting proposed in the guidelines with minor modifications following the reintroduction of descriptions in the ‘local level monitoring’ section.
• Aid and beyond: Some respondents asked whether the Index just covered Official Development Assistance (ODA). As described in the Technical Paper in 2016, the Index assesses overall levels of transparency from major donors and covers all forms of development finance, not just ODA. We do not think this merits changing the name of the Index, but will review the language we use to make it clear that it goes beyond aid.
• Sampling: We have reviewed the approach we take to sampling through the Aid Transparency Tracker (sampling on particular indicators is one of the ways that we test the quality of the information provided in the IATI format). For example, we explored whether we could sample a broader range of countries in an organisation’s portfolio. However, this is difficult to do as not all publishers specify the countries or regions where they operate. Instead, we will double the minimum number of documents required to pass sampling. We will also restrict sampling to current activities, and will add project titles and descriptions to the information sampled.

Further work needed

We also heard through our consultations that there are some areas that require further work. They are:

• Commitments to aid transparency: Most respondents agreed that IATI implementation schedules were no longer a useful measure of commitment to transparency, as they were designed to meet the 2015 deadline in the Busan Declaration. However, there was no consensus on what should replace them. Several respondents suggested that we look at the mechanisms that providers of aid and development finance use to engage with civil society. We will explore ways to measure this, looking both at policy and practice.
• Data use. We consulted on different ways to measure the extent to which publishers, partner country governments and civil society organisations are using the data assessed in the Index. Most respondents agreed that it would be difficult to measure data use systematically, and advised that we should conduct in-depth work on data use at country level. We are now exploring how to do this, separately from the Index work, building on work already done by others.
• Scoring and thresholds for publication. A large majority of our respondents supported the principles of having quality and visibility thresholds that we proposed for IATI data. Most of the questions raised on the thresholds concerned how to implement the visibility threshold, the objective of which is to encourage organisations to publish a large proportion of their portfolio in the IATI standard. We agree that this needs further attention.  We are working on an alternative proposal that relies on publishers self-declaring expected spends on development-related activities for a given year, and we will have more discussions as part of this consultation about this.

Next steps

We want to raise the bar, and we want to give publishers time to raise their game as well. As such, the process to finalise the Index methodology will take some time. We will begin consulting on the issues above in February and March, and announce a timeline for data collection and publication for the next Index by April 2017. This will be followed by a technical paper detailing the new methodology for the next Index.

In the meantime, Publish What You Fund is pleased to announce that we are developing a tester based on our Aid Transparency Tracker. The intention is to allow publishers to self-assess their IATI data files against the Tracker tests. We expect to present a beta version of the tool at the IATI TAG meeting in Tanzania in March. We will continue to update the tester, along with the Aid Transparency Tracker, following the technical consultation on the Github platform. Publish What You Fund’s team looks forward to discussing the tester and continuing the consultation on the Index methodology with the aid transparency and development community.