In this blog we highlight a crucial part of the Aid Transparency Index assessment: the independent reviewer role. The 2022 Index process, which started on 15^th November last year, has so far involved three months reviewing several thousand documents, large amounts of International Aid Transparency Initiative (IATI) data, 50 publishers and 36 independent reviewers. The independent reviewer role supports the manual survey part of the Index process. The manual survey allows us to assess the full portfolio of a publisher’s data by consulting other sources, such as an organisation’s website, data portal or open data policies. The independent reviewers provide an external expert opinion from someone who is independent from both Publish What You Fund and the organisation being assessed.

To recognise the voluntary work of our independent reviewers we recently interviewed four long-standing reviewers who, between them, have provided over twenty years of support to the Index: Amy Dodd from The ONE Campaign; Paola Palacios and Vania Montalvo from Transparencia Mexicana and Sarah Rose from Center for Global Development (CGD). We asked them to tell us why they continue to support the Index:

“It’s a useful resource in our work and an important tool to help us hold governments and institutions to account on transparency asks.”
Amy Dodd- ONE Campaign

“I’ve been happy to support the Index for as long as I have. While policymakers and dedicated civil servants are still working hard to implement—and advance—their transparency commitments, there is, in many ways, less of a focused political spotlight on the transparency imperative now. The Index provides an important source of continued advocacy for transparency and encourages continued progress.”

Sarah Rose – CGD

“We support the Index since it is a tool that allows the public to compare the transparency of the main donor agencies around the world in an independent way. Transparency and accountability should be applied to donor agencies too since the funds usually come from taxpayers in different countries; it is necessary to be able to track and follow the money from the origin to the final destination.”
Paola and Vania – Transparencia Mexicana

Who are our independent reviewers?

The independent reviewer role has been a part of the Index methodology since its inception in 2010 when 17 peer reviewers provided support for the assessments of 30 aid organisations. As the number of organisations in the Index increased, so too did the number of reviewers. For the current 2022 Index we have 36 reviewers from 31 diverse organisations including universities, advocacy organisations, multilateral agencies, NGO networks and independent researchers. Many reviewers have provided consistent support over the years.

What insights do our reviewers have on how the role is important for the Index?

“It gives it a sense of objectivity and that there is an expert sense check on the findings.  It also helps to ‘buy in’ other key stakeholders to the Index itself.”
Amy Dodd- ONE Campaign

“There are times when the scores assigned by the publisher and the Publish What You Fund team don’t align. The independent reviewer is a helpful third perspective that can help evaluate both submissions and help determine which seems more accurate.”

Sarah Rose – CGD

What does an independent reviewer actually do?

The Index assessment begins with an initial download of organisations’ IATI data. However, not all organisations have IATI data, or indeed publish all of their data to the IATI standard. If an organisation has published little or no data in the IATI format, the Index scores them based on a manual survey of information published on their website or data portal. In the 2020 Index 64% of assessed publishers relied on the manual survey section for a quarter or more of their total scores, with 19% relying on the manual survey for over half of their scores. For example, China and Turkey, which are not currently registered as IATI publishers, are assessed entirely in the manual survey.

The results of the manual survey are also shared with each organisation to ensure they are given a chance to comment. Before the results are finalised the survey, along with any comments, are passed over to the independent reviewers. On the 10^th January 2022 36 reviewers were given two weeks to assess data from additional sources from between three to 37 indicators (depending on the extent to which an organisation has published IATI data). Reviewers take into consideration organisations’ comments and adjudicate on discrepancies between research findings.

What do the reviewers get out of assessing a given publisher’s data?

“To me, re-reading the PWYF survey definitions each year provides a helpful reminder of the transparency commitments donors have made and the expectations we should have of donor agencies’ reporting. Especially the first time I review an agency, I’ve been pleased to see links to sources of data—provided as part of the donor or PWYF reviewers’ comments—that I hadn’t previously known about.”
Sarah Rose – CGD

“Having a periodic review process has helped us analyse the progress or changes in the donor’s data and how it is presented. It improves our understanding of how transparency is viewed by donor agencies in comparison to governments and the need to push for better and consistent transparency on both sides.”
Paola and Vania – Transparencia Mexicana

Why do we have the reviewer role?

The role contributes to the impartiality of the research. Reviewers are often specialists in the aid and development policies of the organisations they review. Therefore, they provide a crucial bridge when it comes to reviewing multiple sources, particularly where a language barrier is concerned. They ensure that we have captured an organisation’s full transparency data from websites or other sources.

What have our longest standing reviewers noticed about how the Index has impacted publishing practices?

“[There is a] more systematic, clear way of reporting on their websites.”

Sarah Rose – CGD

“Throughout the years, international trends regarding transparency and accountability have been included in donors’ information, for example, open data. The demands for more transparency around the world to reduce the risks of corruption have also touched donor agencies.”

Paola and Vania – Transparencia Mexicana

But it’s not all been positive, in recent years, some reviewers have noticed a decline in publishing practices with data becoming less available demonstrating that an organisation’s scoring in the Index can go down as well as up.

The independent reviewer role improves reliability and validity of the Index survey results and provides an opportunity for organisations that use open aid data to have a say in the assessment of the data they use. We would like to thank all of our reviewers for their invaluable contribution to the rigour of the Aid Transparency Index.