Methodology

For data-gathering purposes, the Revenue Watch Institute developed a detailed questionnaire to identify information about a government’s management of the extractive sector and define revenue transparency. This questionnaire is based on the standards put forward by EITI, the PWYP campaign and the IMF’s Guide on Revenue Transparency.

The questionnaire assesses information that governments publish about the oil, gas and mining sector in a comprehensive manner. First, the questionnaire identifies a set of key documents published by governmental agencies that oversee extractive resources. These documents provide information about reporting practices and constitute a straightforward standard to compare different countries. Second, the questionnaire identifies publicly available information and institutional practices on a series of issues highlighted by international guidelines, experts and international campaigns on transparency.

The index is based on the collection of information covering seven categories that represent key areas of natural resource governance:

  • Access to resources: availability of data detailing contracts and licensing procedures, as well as the existing legal and regulatory mechanisms related to the accessibility of information.
  • Generation of revenue: the availability of detailed information published by various government agencies on production and payments as well as an assessment of its accessibility and periodicity.
  • Institutional setting: the rules, roles and responsibilities of the main actors involved in the management of revenue generation, as well as the presence of internal controls and other checks.
  • State-owned companies (SOCs): the availability of information regarding the governance structures of state-owned entities and the reporting practices related to their activities.
  • Natural resource funds: the rules governing the operation of funds and the availability of information regarding their functioning.
  • Sub-national transfers: the laws that regulate resource revenue sharing between different administrative and political units and assess the existing reporting practices in this regard.
  • EITI: examines the extent to which members have fulfilled EITI criteria (i.e. publication of EITI reports, independent payment audits and reconciliations, and information about payments and revenue from state-owned companies).

The questionnaire distinguishes transparency aspects from the legal and regulatory framework. The transparency component refers to whether governments publish information relevant to each of the seven categories, as well as how comprehensive it is and how frequently it is published. The legal and regulatory framework dimension refers to the laws, regulations and institutions that delineate roles and responsibilities in the extractive sector and provide assurances of integrity in relevant categories.

Based on this questionnaire design, the Revenue Watch Index is constructed as a simple average of the transparency-related questions. The scoring of each question is based on whether a document, regular publication or online database provides the information demanded in the questionnaire. Using only this type of question allows the index to compare whether information exists in periodical reports and documents, a straightforward way of establishing standards across countries.

For the legal and regulatory framework component, we draw from standards identified by the IMF guide and the Natural Resource Charter, among others, to select practices that facilitate comprehensive disclosure, offer greater access to information, or put a check on discretionary powers. The rationale for including these questions springs from the recognition that transparency alone does not address all the problems faced by resource-rich countries. However, standards and good practices are still emerging and evolving. In some cases, policy recommendations can become prescriptive. Therefore, the index scores rely exclusively on transparency questions.

Questions related to the legal and regulatory framework track the existence of rules and organizational features, but do not assess how they are functioning in practice. The questions are meant to help civil society and parliamentarians further evaluate domestic institutions and improve their performance. In a separate section of this report, we provide a brief analysis of the legal and regulatory framework features using the information researchers collected. The index does not score the legal and regulatory framework, and the information gathered through these questions is useful to provide context for the countries included in this report.

2.1 Country coverage

Country selection began with the 55 countries that the IMF defines as having economies dependent on oil, gas or minerals. In addition, we considered countries that participate in the EITI. At the time this project started, this included 30 countries, all but six of which are in the IMF definition. Finally, we identified five important producers of hydrocarbons and minerals that are among the top 10 producers of mineral commodities, but did not fall in either of the two groupings (i.e. Australia, Brazil, Canada, China and the United States). These three categories produced a potential list of 66 countries.

For a balanced geographical selection, we identified six regions: (1) Africa, (2) Asia-Pacific, (3) Central Asia/Russia, (4) Latin America, (5) Middle East and North Africa (MENA) and (6) Organization for Economic Co-operation and Development (OECD) countries. From these, we selected a sample of 41 countries out of the potential list of 66 for the first implementation of the Revenue Watch Index. Despite a nonrandom selection, the countries we analyze provide a balanced regional approach for comparative purposes. For this report, countries with a federal structure (e.g. Canada and Australia) were left out due to the challenges posed by the diverse nature of the authorities in charge of managing and taxing natural resources in terms of research and creating an aggregated score.

2.2 Research process

Independent consultants gathered the information to complete each country questionnaire from November 2009 to April 2010. Research concentrated on identifying publicly available information covering a period from January 2006 to December 2009. Reports published during this time but providing information prior to 2006 counted as historical data. A number of countries implementing the EITI produced new reports in 2010, and the disclosure of new information through such reports is likely to improve country scores in the next iteration of the index.

Each researcher covered countries within a particular region, of which they had expert knowledge (including relevant local languages) and prior work experience. Researchers filled in a detailed questionnaire for each country and provided evidence for their answers. Research was based on publicly available information, which we define as information that any and all citizens might be able to obtain on an official Internet website or through a request to the public authority issuing the document (see “Key Documents”). If one or more of the key documents could be obtained only through means unavailable to the public, it was not taken into account in scoring the questionnaire. In addition, researchers conducted interviews, to the extent possible, with local experts, civil society activists and government officials to corroborate information, expand our sources of information and mitigate any technological or translation bias.

It is important to note that information published on Internet websites by governmental agencies is the main source of data for the index. This is because the purpose of the index is to test how much data governments make publicly available through channels easily accessible to citizens. All the governments in the survey maintain official websites and routinely publish relevant information on them. A preliminary test of our methodology in Angola, Ecuador, Gabon, Mexico, Norway, Peru and South Africa captured a significant percentage of relevant information from online sources and identified the agencies more commonly in charge of reporting.

2.3 Resource focus

The Revenue Watch Index includes 30 countries where oil and gas are produced and 11 where minerals are extracted. For the mineral producers, researchers sought to identify information covering the entire sector. However, while oil and gas are relatively standardized, mining is a complex sector producing numerous resources that have different metrics for calculating volume and value.

Analysis is further complicated by the fact that some hydrocarbon-rich countries are also important mining countries. In order to make comparisons across diverse countries, this research concentrates on the resource that earns the most revenue for the government as defined by the IMF guide. This decision means that our results do not cover the totality of the extractive sector in every country. Nonetheless, our findings and conclusions are valid for the resources that contribute the most to the government’s fiscal income in the majority of countries reviewed here.

2.4 Peer review

The research results were submitted to peer reviewers to verify the answers provided in each questionnaire. Every country questionnaire was reviewed by at least one expert. Most peer reviewers were local experts familiar with the revenue transparency movement and with expertise on the specific country’s extractive sector. Reviewers provided comments, suggested changes, challenged interpretations, provided overlooked evidence and/or identified mistakes. The resulting questionnaire was then sent back to RWI, which confirmed that all peer reviewers’ comments followed the methodology and were consistent. All the comments were subsequently shared with the lead researchers, who responded to reviewers’ comments. If the reviewers’ evidence or suggestions led to changes in the answer choice for specific questions, researchers made changes and appropriate notes about any modification, mistake or correction. RWI staff checked for consistency of assumptions across countries when selecting scores and made final decisions on scoring. Finally, RWI collected information on the number of changes suggested by each reviewer and number of question scores changed after the reviewing process. We provide these as an indication of the discussions and the decision-making process behind the final calculations of the score (see Appendix 3). Peer review started in February 2010 and was completed by July 2010. The result of this process is an expert-based survey index, in which every answer is backed up by evidence as defined before.

2.5 Scoring and organization of results

As mentioned above, only transparency-related questions were factored into the Revenue Watch Index. The index follows the seven categories that compose the questionnaire: Category I Access to resources; Category II Generation of revenue; Category III Institutional setting; Category IV State-owned oil, gas and mining companies; Category V Natural resource funds; Category VI Sub-national transfers; and Category VII EITI.

The most important section for the score is Category II on Generation of revenue, with 29 indicators, compared with 22 for all these other sections. The Revenue Watch Index questionnaire repeats five sets of questions covering information from the finance and extractive sector ministries, regulatory agencies, central banks and state-owned companies (or agencies performing relevant role in each case). This design stressed the need for comprehensive research of all the potential sources of information in every country. For the final score of this section in the Revenue Watch Index, we systematically chose the maximum value for each indicator from among the five available sets. This consolidated score avoids rewarding repetition of information and it forms the core of the index.

Indicators for Category II highlight crucial elements of a country’s fiscal system for the extractive sector; what types of payments governments receive for their oil, gas and minerals, and in what amount. The index captures all possible forms of payments, including royalties, special taxes, excise taxes, percentages of revenue share in production sharing agreements and others.

Each question was scored on a scale of 0 to 100, reflecting the variable availability of information to the public. The final score is a simple average of all the values of the transparency-related questions. All questions (multiple-choice and yes/no questions) have a “not applicable” option as an answer. Scoring questions as “not applicable” was avoided in the index whenever possible. We used it only when the situation was truly not applicable, such as in response to questions about the existence of a state-owned company in a country that does not have one. If a question was scored “not applicable” by a researcher, the peer reviewer had to agree with the score, and RWI checked the evidence available before the score was final. “Not applicable” questions were dropped from the pool of questions used to determine a country’s score in the Revenue Watch Index.

Based on the final results of the index, we identify three groups of countries, ranked according to their relative score out of a possible 100:

  • Comprehensive Revenue Transparency (average score 67-100): countries in this group provide their citizens with substantial amounts of information regarding the management of their extractive sector. Their governmental agencies and/or SOCs show strong reporting practices and tend to make available detailed or disaggregated data on the different areas under their authority.
  • Partial Revenue Transparency (average score 34-66): countries provide their citizens with information about the extractive sector yet show important transparency gaps in one or more specific categories of the index.
  • Scant Revenue Transparency (average score 0-33): countries have the lowest levels of information disclosure and poor reporting practices all across the different categories that compose the index.

The results vary greatly among the countries included in the index, and the questionnaire is helpful in highlighting the specific areas where each country falls short of good practice. Therefore, even for countries with Comprehensive Revenue Transparency, a closer examination of their scores for specific indicators within each category demonstrates substantial room for improvement.

2.6 What the Revenue Watch Index does not cover

This index is not a measure of corruption or budget openness. There are other well-known and firmly established organizations that already produce reports on these issues. Readers interested in those topics should refer to Transparency International’s Corruption Perception Index, the International Budget Partnership’s Open Budget Index (OBI), Global Integrity’s Global Integrity Report and the World Bank’s World Governance Indicators, among others. In contrast, this index provides a new measure for a specific sector that has been overlooked by comparative governance indicators until now.

In a similar vein, the index does not measure whether a country’s extractive sector effectively contributes to economic growth and development. Neither does it focus on the accuracy or completeness of the information disclosed and identified. Countries that may engage in unreported off-budget accounts would not be identified by the methodology followed in this study. Therefore, any claims on the basis of this research should be limited to the information we can support with primary documents and reports. Along these lines, future research for the index may test the reliability and accuracy of information that official sources provide, which could involve gathering quantitative data and cross-referencing with international or independent sources.

This index creates a baseline that enables country comparisons and provides a tool to assess whether and how countries are seizing opportunities to make their extractive sectors more transparent. The comparative nature of this index, as well as its specificity at the policy level, also provides a powerful tool for policy makers to assess their institutional performance and for local advocacy partners to articulate concrete areas in need of reform.

No single measure can capture all the dimensions of a country’s institutional practice and governance. However, the Revenue Watch Index provides the best information available under the assumptions and rules explained in the methodology. This index is not a comprehensive overview of a country’s governance—but it measures indicators for a specific sector that has been neglected until now.

Using a similar standard as the International Budget Partnership (IBP)’s Open Budget Index, we define publicly available information as information that any and all citizens might be able to obtain through a request to the public authority issuing the document. During the information-gathering process, researchers inquired about the following documents:

  • Contracts or investment agreements: agreement between a state (or any of the authorized agencies acting on its behalf) and resource companies that regulate access to resources. These are normally published by the ministry of the extractive sector or the SOC.
  • Environmental and social impact reports: assessments including information about the environmental and social impact of activities undertaken in the framework of the extractive sector. These tend to be published by the SOC or ministries of environmental protection.
  • Statistical reports/databases: documents containing statistical information related to the extractive sector. In general, statistical reports are produced and published by the ministry of the extractive sector, statistical, and regulatory agencies or by the SOC.
  • Annual and in-year reports: annual and/or quarterly or monthly documents describing the activities and operations of a country’s government with regards to the extractive industry. While these are usually published by the government agencies, in some countries the central bank also publishes reports on its activities regarding the extractive sector.
  • EITI reports: documents containing data of revenue flows from oil, gas and mining activities of a country. Normally an independent EITI report administrator writes these, and the country’s multi-stakeholder group accepts them.
  • Auditor reports: either an annual report issued by the Supreme Audit Institution (SAI) attesting to the government agencies’ yearend final accounts and reports from internal or external audit agencies that provide audited financial statements for the SOC or other agencies managing resource revenue.

2.7 Key documents identified by research on the Revenue Watch Index

The research for this index identifies a host of valuable information and documents. Primary official documents are identified as “Key Documents,” and these are the main evidence on which the Revenue Watch Index bases its scores. Definition of these documents relies on the IMF guide, EITI and other relevant international standards.

In addition to collecting key documents, the Revenue Watch Index research identifies the degree of public access to key documents governments provide. Table 1 (Index p. 14) summarizes this information. Documents provided by researchers are evidence of countries actually producing and distributing key documents to the public. Our inference for the other three categories in the table relies on interviews with local experts and input from peer reviewers.

Table 1 demonstrates that contract transparency is very rare. Only five governments publish contracts on mineral resources in full. In contrast, annual and statistical reports from ministries of finance or oil and mining are fairly common. Regarding environmental and social impact reports, the majority of governments do not publish documents incorporating this type of information.