It’s Friday night and you have left it a bit late to find a restaurant to meet your friend at. After a quick search online you come across an Italian eatery at a good half way point. However, when you arrive the worst has happened: it’s closed, has been turned into a bubble tea shop or the quality isn’t what you expected. Perhaps the opening times were incorrect or worse, you were misled by some reviews. Your trust in the data has been eroded and the tools which we built to keep us informed are no longer working.

It’s hard to put a figure on what percentage of online data is defective, but there’s no doubt it’s a real problem. There has been a lot in the news recently about how tech companies can ensure the information they are publishing is accurate, up to date and transparent. It’s become such an issue that rating tools such as Fakespot have now been built to help the discerning shopper get a better sense of the accuracy of online information.

Data accuracy is just as important for international development work. However, instead of selling a product or service, major aid donors are publishing data on billions of dollars of Official Development Assistance (ODA) which is distributed into developing countries every year. It would be relatively straightforward if it was just one organisation publishing information about international aid, but the constellation of aid data stretches across numerous donors, multilateral organisations and implementing agencies each with their own approach to information management. Like Fakespot, an independent third party can be your answer to ensure quality data. At Publish What You Fund, we have been watching the data quality of aid data since we were established in 2008.

The Index

We are now at the midpoint of the 2020 Aid Transparency Index. The Index tracks and assesses the progress towards transparency made by the world’s major aid donors. This year we are assessing 47 major aid donors including bilateral donors from Europe, the US and Asia, multilateral institutions and development banks. Comparatively scoring and ranking these donor agencies, based on a robust methodology, enables us to identify changes needed and to galvanise major donors to progressively increase and improve the aid and development information they make available.

At the beginning of December 2019 we collected a first cut of the donors’ data published in the International Aid Transparency Initiative (IATI) standard and it is this data we are now working with. First we used our Aid Transparency Tracker software to score donors based on the frequency and consistency of their published data. It’s now time for the real quality checks as we manually sample the data for quality.

Sampling

During sampling we randomly select 20 samples from each donor’s data for 16 of the indicators scored in the Index. The team at Publish What You Fund then reviews the documents or data to see if they meet the required quality standards. The purpose is to ensure that the information published is of a sufficiently high quality, that it is current, the right type of document has been provided and that it is consistent with the rest of the documentation for that activity. By taking a random sample of 20 documents we can estimate to an acceptable level of confidence what proportion of data in the organisation’s total data set meets our quality standards. For an indicator to pass sampling at least half of the sample needs to be accepted. Any data that did not pass our sampling checks are flagged with the organisation so they know what areas they need to focus on to improve. Sampling generally yields good results, for the 2018 Index we saw a 13% improvement in the average score after feedback from our sampling.

So far we have sampled over 6,000 documents and data points across the publishers and indicators, ranging from sub-national locations to project evaluations and implementing partner contracts. We have noticed that there has been an increase in project description data – six donors have added this data set to their data since 2018, and three more donors are now publishing their project objectives, which is great news! Organisations do have a second chance to pass our sampling checks when we make a final data cut at the end of March and when a second round of sampling will take place.

Manual Survey

The next stage of the Index involves a manual survey where we search for any data which organisations are not publishing to IATI or if data has not passed the sampling checks. Publish What You Fund maintains regular communication with the organisations who are included in the Index. From the first data cut to the last, we provide regular updates on their transparency such as any broken links, and out-of-date or inconsistent information. This is so the organisations have plenty of time to amend and improve their data.

We also have 39 independent reviewers who have a say in the scoring process. These foreign aid experts make an impartial assessment of research and weigh in on the accuracy, relevance and coherence of the data and scoring. The independent reviewers are now assessing our findings for each of the donors and providing their expert opinions.

The importance of trust

The Aid Transparency Index has proven to be an effective tool to encourage donors to publish quality aid data because of the sampling process. After a final data pull in March the total scores and ranking will be released in June and we will then see which donors have shown the greatest improvement in providing accurate reliable data that we can trust. There are several other tools which are driving improvements in IATI data quality and reliability including the new updated IATI validator and Publish What You Fund’s IATI canary tool.

Building and maintaining trust in our aid system has never been more important; ODA is currently coming under scrutiny both in the UK, with the new government looking at a departmental re-structure, and in the US with President Trump’s impeachment inquiry hinging on the use of aid. Making aid policies, financial data and project information public goes some way to building trust. But, just like other online search tools, the information must also be accurate, up to date and of high quality if that trust is to be maintained.