1. What is the Aid Transparency Index?

The Aid Transparency Index (hence forth ‘the Index’) assesses the state of aid transparency among the world’s major donor organisations. The Index tracks and encourages progress, while holding donors to account.

2. In what ways is this year’s Index different from last year’s?

The main difference with last year is that the 2016 Aid Transparency Index focuses on fewer and bigger donors as well as those that are instrumental to advancing the course of aid transparency. The addition of the UAE in this year’s Index is also a key change in moving the aid transparency issue beyond the Busan agenda and looking at emerging donors.

3. How do you choose which organisations to include in the Index?

Organisations have been selected using three criteria:

• They are a large donor Official Development Assistance (ODA) spend is more than USD 1bn);
• They have a significant role and influence as the country’s major aid agency and engagement with the Busan agenda;
• They are an institution to which government or organisation-wide transparency commitments apply, for example members of the G7 or all U.S. agencies.

Organisations need to meet a minimum of two of these criteria to be included in the Aid Transparency Index. There are some donors that are spending more than USD 1bn in ODA that have not been included, for example India, Saudi Arabia and Turkey, even though others spending lower volumes of ODA have been retained. This is because the latter are covered by government or organisation-wide transparency commitments (e.g. U.S. government-wide commitment to aid transparency). Although we would like to include more donor organisations in the Index, this is not possible currently due to resource constraints.

4. Have you added or dropped any organisations in this year’s Index?

The list of organisations included in the 2016 Index has been revised. Having reviewed the criteria for donor selection it has been decided that the 2016 Index will concentrate on fewer, bigger donors, as well as those that are instrumental in advancing the course of aid transparency. The 2016 Index ranks 46 agencies – 45 of the donors that were included in the 2014 and 2013 Indices have been retained. The United Arab Emirates has been included for the first time due to the increasingly influential role being played by the UAE in international development and the importance of transparency in the post-2015 Agenda.

5. Why do you select more than one agency for some donors?

The Aid Transparency Index assesses more than one agency for some large donors (EC, France, Germany, Japan, UN, U.S. and the World Bank) with multiple ministries or organisations responsible for significant proportions of ODA. We have opted to maintain the disaggregation of agencies for several reasons. First, no two agencies from the same donor country or organisation in the Index score the same. There is often wide variation in the amount of information made available by different agencies in a single country or multilateral organisation. Second, agencies often retain a large amount of autonomy in deciding how much information they make available and have different publication approaches, and should therefore be held accountable for them. Third, it would be unfair for high performing agencies within a country or organisation to be pulled down by lower performing agencies, and similarly lower performing agencies should not have their poor performance masked in an average score. Finally, it is unclear how we can aggregate agencies into a single country or organisation score in a way that reflects wide variations in performance. It would be necessary to take into account the proportion of a country’s aid delivered by each separate agency in order to create an aggregate country ranking that fairly reflects that country’s level of aid transparency and this information is not always available.

6. Why are some donor agencies assessed jointly with others?

The Aid Transparency Index is primarily an assessment of activity-level information provided by large or influential agencies that are covered by transparency commitments. Where a ministry or equivalent parent organisation, distinct from an implementing agency, is responsible for funding, strategy or policy-making for the implementing agency, we look at information from both organisations. The resulting assessment often bears the name of both agencies assessed. For example, the Ministry of Economic Cooperation and Development is jointly assessed with its two major implementing agencies, GIZ and KfW. The resulting assessments are labelled BMZ-GIZ and BMZ-KfW respectively. In other cases where a ministry undertakes direct implementation, we separately assess them. For example, for Japan we include separate assessments for the Japan International Cooperation Agency and the Ministry of Foreign Affairs.

7. Do you rate donors that fund you?

The same methodology applies to all donors included in the Index. Funders can be included if they meet the set of criteria established to select donors for the Index. The complete list of Publish What You Fund’s funders is available at www.publishwhatyoufund.org/about-us/funders-2/

8. What indicators does the Index use? How many are there and how are they weighted?

The 2016 Aid Transparency Index uses 39 indicators grouped into weighted categories, to assess how transparent donor organisations are about their aid activities. These categories cover overall commitment to aid transparency and publication of information at both organisation and activity level. Within the publication category, the organisation-level indicators account for 25% of the overall weight, while the activity-level indicators account for 65%. The two publication groups are further divided in subgroups, based largely upon the subgroups used in the Common Standard implementation schedules template. The subgroups are equally weighted.

9. Have any indicators from the 2014 Index been dropped? Have any new ones been added?

All 39 indicators used in the 2014 Index have been retained in 2016. No new indicator has been added.

10. How do you gather information on what information each donor is publishing?

Most information is gathered from what is published online by each organisation – either on their website, on the IATI Registry or on national data platforms such as the U.S. Foreign Assistance Dashboard. Two indicators use secondary data sources, to assess the quality of Freedom of Information legislation and donor’s implementation schedules.

If the organisation is not an IATI publisher then all the information is collected via the manual survey. For organisations that are publishing to the IATI Registry, data collection follows a two-step process:

• First, their data is run through the data quality tool of the Aid Transparency Tracker, which is designed to run automated checks and tests on each organisation’s data, providing both a comparative view across organisations and granular details on each organisation’s data. These tests are aggregated to produce scores for indicators to which they are relevant.
• Next, for those indicators for which information is not published to the IATI Registry or does not pass the necessary tests, the data is collected via the manual survey.

All organisations are provided with an opportunity to review the assessments and provide us with any feedback for consideration. Surveys are also independently reviewed.

11. Once data collection has started, do you advise donors individually on how they can improve?

Yes, we are always willing to engage with donors and discuss how they can increase their transparency. As well as some common issues, donors tend to have unique challenges to increasing their transparency (such as internal reporting or knowledge management systems). We are happy to provide information and support to donors. All donors will have an opportunity to review the data we collect for them and can provide us with clarifications and corrections as needed.

12. How did you select independent reviewers/CSO partners for the Index?

We usually work with experts and national NGO platforms for aid effectiveness and development. For most of the EU member states, we approach the AidWatch/CONCORD platform, which then recommends members to us.

If the platform members are unable to conduct the review, we ask them to recommend other organisations to us. Where there is no national NGO platform, we work with CSOs we have partnered with in the past on the Index or in other advocacy efforts. For multilateral organisations or IFIs where there is no direct match with an NGO platform or CSO, we ask our peer reviewers to provide recommendations on who we can approach for the independent review. The independent review process is voluntary and unpaid. There are some organisations for whom we are unable to find independent reviewers. In these cases, Publish What You Fund undertakes the assessment.

13. How are you sampling documents and data on results, conditions and sub-national location published to the IATI Registry?

A total of 14 indicators refer to documents. These documents are manually checked to verify that they contain the required information to score for the indicator. A minimum of five documents need to meet the required criteria to score for the indicator. For IATI publishers, the documents will be randomly selected from those projects that pass the tests for the relevant indicator. Data published to the IATI Registry on results, sub-national location and conditions will also be sampled to ensure it meets the criteria for those indicators.

14. What is the Aid Transparency Tracker?

The Aid Transparency Tracker is an online data collection platform that provides the main, underlying dataset for the Aid Transparency Index. The Tracker includes three separate data collection tools:

• An automated data quality assessment tool (for indicators where comparable and timely data is available via IATI).
• A survey (for indicators where comparable and timely data is not currently available).
• An implementation schedules’ analysis tool.

15. What do you mean by “format” of the data?

There is a substantial difference between searchable IATI XML data published to the IATI Registry where you can access and compare any number of worldwide projects across a number of fields as opposed to searching dozens of URLs or looking for information published in several different PDF files. This difference has been quantified by using graduated scoring approach for 22 indicators depending on the format of publication.

16. How did you score data formats? Why are 22 indicators scored on format and others not?

There are 39 indicators in total, of which three measure commitment to aid transparency and 36 measure the publication of information. The scoring methodology for the publication-level indicators takes into account the comparability and accessibility of information. For 22indicators, data published in PDF format scores lower than data published in Excel, CSV or IATI XML formats. Data that is published in the most open, comparable format of IATI XML and is available via the IATI Registry can score up to 100% for certain indicators, depending on quality and frequency of publication.

For 14 other indicators, the scoring approach recognises that format is not so important – an annual report published in PDF is much the same as an annual report published on a webpage. However, the inclusion of links to such PDF documents in an organisation’s IATI data is more valuable – especially at the activity level – as it makes them easier to locate and identify than documents available just on the organisation’s website. Therefore documents made available via links through IATI are scored higher than documents available through other sources.

17. Why is IATI so important? Aren’t other forms of publication just as good in their own way / for specific purposes?

IATI is the agreed standard for publishing current aid information in a common, comparable format. While donors may publish extensive information on their own website or to the DAC, it will always lack these vital elements of being current and comparable. IATI also includes some “added-value” fields, for example results, impact appraisals, sub-national location and a budget identifier.

18. How do you measure the quality of IATI data? What are the tests you run on IATI data?

The quality of data published in IATI XML is assessed by running a series of tests on all activity and organisation data packages being published to the Registry. These tests have been designed to assess the availability and comparability of aid information and to determine whether an organisation’s IATI data conforms to the IATI Standard appropriately. Most of the tests have been derived directly from the IATI schemas which provide formats for reporting data on various information fields to the IATI Registry. The tests return results for the percentage of activities within an organisations’ data packages that contain correctly coded information on the specific indicator being tested. For example: what percentage of activities reported contain a title? Or what percentage of activities that have been completed contain information on results? The full list of tests can be accessed here.

19. Why does IATI XML data have to be on the IATI Registry for it to be taken into account in 2016?

Only data linked to the IATI Registry will be taken into account in 2016 in recognition that it is easier to locate and use than data published in lots of different locations. The IATI Registry is an important component of IATI publication, as it makes data discoverable and easier to access. IATI publishers “register” their IATI XML data, providing links back to the original source data – which remains on donors’ own websites – and other useful metadata. As a result the IATI publication approach is interpreted strictly for the Index, meaning that IATI XML data needs to be available via the IATI Registry for it to be taken into account. IATI XML data that is not on the Registry is scored the same as other machine-readable data.

20. Why has there been more emphasis attached to the frequency of publication to the IATI Registry since the 2014 Aid Transarency Index? Why is frequency of publication only assessed for IATI data?

In 2014, we adopted a more nuanced approach to frequency, so we can monitor if donors are publishing monthly, quarterly or less frequently, in line with the Common Standard commitment and partner countries request for data at a minimum on a quarterly basis. This change was made based on Publish What You Fund’s public consultation on the Index data quality tests and based on the findings of a survey of the needs of Aid Information Management Systems (AIMS) used in partner countries. Only IATI data is scored on frequency. Unfortunately, it is not possible to take into account frequency of publication for data published in other formats because the information is not always time-stamped.

21. Why do activity budgets have to be forward-looking and broken down by quarter for the first year ahead for IATI publishers to scre the maximum available points on indicator 33?

This is based on partner country feedback on a country AIMS survey, conducted by IATI, which emphasised the need for forward-looking data that is broken down by quarter for planning purposes. Providing annual forward budgets allows an IATI publisher to score up to half the total available data quality points, while a quarterly breakdown for the first year ahead will enable them to score the remaining half (information published to IATI is scored higher than information published in other formats).

22. Who actually uses IATI data?

Now that an increasing amount of information is being published to IATI, the challenge is to encourage wide-ranging use of the data. At Publish What You Fund, we use IATI data in the Index to measure whether and to what extent donors are delivering on their promise to make their aid transparent. The case studies developed in the 2014 and 2016 Index reports as well as the 2015 EU and U.S. Aid Transparency Reviews highlight examples of data use. For instance, the Netherlands is using its own information for internal management and reporting purposes and is working with two of its partner organisations publishing to IATI in order to stimulate exchange and learning, with the longer-term aim of including open data throughout its supply chains. The pilot studies on data use by partner countries, conducted by organisations such as USAID highlights existing data gaps and ways in which information can be made more useful. The Aid effectiveness project in Bangladesh is one of the most recent examples of IATI data import into Aid Information Management System of a partner country. Similar initiatives have or are taking place in other countries such as Senegal or Cote d’Ivoire. Together, these initiatives hold great promise for unlocking the potential of IATI.

23. Are there any visualisations of IATI data?

A number of organisations are now using open data platforms driven by IATI data, marking an important shift from publishing raw data to visualising it in a meaningful way for users. In addition to the ones highlighted in the 2014 report, a number of organisations have implemented recent changes, notably using an open license on their portal, therefore allowing for use, reuse and modification of the data. Changes were implemented in particular by the U.S. Dashboard (beta.foreignassistance.gov/), GAVI (www.gavi.org/results/disbursements/), UNICEF (open.unicef.org) and France-AFD (afd.opendatasoft.com/page/apropos/). For more on tools using IATI data: see Case Study 1, p9 of the 2016 Index Report.