Tripartite Monitoring and Evaluation Framework Consultation – Human Healthcare
We appreciate the opportunity to share this input on the Tripartite Monitoring and Evaluation (M&E) approach to the global action plan on antimicrobial resistance. Given its charge laid out in the Global Action Plan (GAP) on Antimicrobial Resistance (AMR), for the World Health Organization (WHO), Food and Agriculture Organization (FAO), and World Organization for Animal Health (OIE), the Tripartite M&E framework represents a key source of ensuring accountability across the global health community. To ensure accountability, the M&E framework must include indicators that are actionable. To achieve the greatest impact, the indicators should focus on changes in behavior, not just attitude or knowledge. Both the methods behind the M&E framework and the process of monitoring and evaluation must be transparent; the models behind indicators must be made clear to all stakeholders; and mandates and incentives for transparency must be developed. Members of the Antibiotic Resistance Coalition (ARC) have worked together to call attention to both the need to ensure accountability for and between stakeholders based on a clear theory of change and transparent process.
The Tripartite M&E framework is an important opportunity to ensure accountability among the WHO, FAO, OIE, national governments, national and international partners, civil society and the global community. In order to ensure the M&E framework improves accountability, it is important to consider what parties are being held accountable, to whom the parties are being held accountable, and by what means they are being held accountable. It appears that the draft M&E framework would apply to Member States and to international and national partners, but not to the WHO, FAO and OIE. It would be important to clarify why this is the approach taken.
The M&E framework noted that the “WHO, FAO and OIE will conduct routine monitoring of their own planned activities and the outputs achieved” and that “The recently-formed Interagency Coordination Group on AMR may establish further reporting requirements or independent monitoring of tripartite progress. (Page 4) Monitoring conducted within the WHO, FAO and OIE of their own planned activities also requires oversight. Though the Secretariat supporting the UN Inter-Agency Coordination Group (IACG) process is reliant on the WHO, FAO and OIE for staffing and expertise, the IACG can play an important role if it can maintain some independence. M&E conducted by the IACG allows for an additional level of accountability, between the IGOs and country governments and across the UN family of agencies. The link between the IACG and the M&E framework and between the National Action Plans (NAPs) and this framework should be made clear and be well defined. Including the IACG in monitoring Tripartite progress, will help fulfill the IACG mandate of providing “practical guidance for approaches needed to ensure sustained effective global action to address antimicrobial resistance.”[i]
Making Indicators Actionable
When reviewing the online consultation for the Tripartite M&E approach to the GAP on antimicrobial resistance, members of the Antibiotic Resistance Coalition noted the need to make indicators actionable. To do so, there should be a clear theory of change behind these monitoring and evaluation efforts. In this way, baseline indicator levels, changes in the level of an indicator in a country or comparisons of the indicator across countries might motivate or direct policy changes, resource allocations, or technical assistance efforts.
Overarching Framework Issues
The M&E framework has strength in its simplicity, which contributes to its feasibility for country data collection. The ARC coalition assessed the M&E framework using key factors including its time horizon, its applicability across differently resourced country contexts, and its alignment with the five strategic objectives of GAP.
Different time horizons apply for the baseline evaluation of NAPs and their starting points, the IACG three-year mandate to report back to the UN General Assembly, and the 5-10-year M&E plan laid out by the Tripartite agencies. The adoption and implementation of the M&E framework depends, in part, on how effectively it complements these various other needs, operating on different timetables, without making extra or redundant work. Analyzing how the selected indicators might be constructed to capture near-term gains, year-to-year measurement, and longer-term gains would be important to undertake. An M&E framework that captures short term gains will be valuable to build and sustain political momentum for meeting longer-term milestones in tackling AMR.
Applicability across differently resourced country contexts
In striving to be universal, the M&E framework faces a strategic choice. Either it could opt to establish a “one size fits all” set of indicators, but in so doing, recognize that some countries might not be in a near-term position to report such data or that the indicators selected for the M&E framework must not rely on having such infrastructure. Or alternatively, indicators could be designed with measurement approaches tiered to the stage of development or level of resources in that country setting. By tiering, indicators might be tracked differently depending on the country’s existing resources or data collection capabilities. A tiered approach might enable broader participation among less well-resourced countries and provide steppingstones to deeper engagement as local infrastructure and capacity grow. Regardless a feasibility analysis should be conducted to understand the capacity of countries to comply with reporting on the proposed indicators and to understand better what data are currently available. This analysis could help guide follow-on technical and financial resources to bridge challenges countries may face in ramping up their M&E framework for collecting data.
In addition to a tiered approach to indicator measurement based on stage of development, different countries face very different challenges from AMR. For example, when looking at a country where there are few veterinarians, indicators measuring their training and education may be less important than targeting farmers and the sources from which they receive technical assistance. Providing a menu of supplemental indicators, above and beyond the core, might give important latitude to differently situated countries to measure what is most important in the local context.
Alignment with the Five Strategic Objectives of GAP
The GAP on AMR offers a useful starting framework around which to align an M&E framework. However, aligning the M&E framework with the five GAP Strategic Objectives comes with potential limitations. Most of the larger AMR outcomes will require enlisting intersectoral support, applying several GAP strategic objectives in concert, or sequencing goals where one is foundational to the next. So aligning indicators around the five Strategic Objectives of the M&E framework risks leaving gaps or overlooking potential synergy where two or more strategic objectives crossover and work in concert. Aligning indicators with the five strategic objectives does not guarantee that indicators are aligned with progress on the outcomes sought for tackling AMR, as the five strategic objectives were not designed to be endpoints unto themselves. In order to understand the gaps and interplay among the five strategic objectives, we propose that various scenarios should be tested and a gap analysis completed. For example, if the goal of “More appropriate use of antimicrobials” (Page 18) were chosen, the key drivers behind use might include 1) contributors to irrational use in the provider-patient relationship; 2) substandard and falsified medicines; 3) mispromotion of medicines; and 4) misalignment of economic incentives along the pharmaceutical value chain, among others. When these key drivers are instead sorted using the GAP strategic objectives, each one falls under a different strategic objective, making it difficult to understand at a glance, the complete picture each indicator is attempting to address. Testing of various scenarios can also help identify shortfalls between the proposed implementation of goals and strategic objectives and the international and domestic support available to meet them. This gap analysis will help direct resources, both technical and financial, to where it can best be leveraged.
Access is represented in the general outcome of “More appropriate use of antimicrobials, with increased access when needed and less inappropriate use.” (Page 18) At the indicator level, it would be important to capture key dimensions of access. At the systems level, access to antibiotics might reflect therapeutic (R&D pipeline), financial (marketplace) and structural (healthcare delivery system) access. Considering the components of structural access, Penchansky’s 5 A’s of health care access include affordability, availability, accessibility, accommodation, and acceptability. Of particular importance among the indicators are availability and affordability addressed in the general outcomes of:
- “B.4: Access to AMs when needed: Proportion of health facilities that have a core set of relevant essential medicines available and affordable on a sustainable basis. (SDG indicator 3.b.3, with Access antibiotics disaggregated), and
- 5: Affordability of key products – e.g. number of countries with price of a course of a specific, generic second line antibiotic is more than 7 days’ wages.” (Page 32)
However, this does not address the availability of second-line antibiotics in hospitals with drug- resistant infections. The general outcome indicator in “B.3: Equity index of access by country (like a Gini coefficient, to be defined)” warrants further development, but is promising as one measure of country-level access to life-saving antibiotics.
Both the methods behind the M&E framework and the process of monitoring and evaluation must be transparent. Mandates and incentives for transparency must be developed at multiple levels including for hospitals and industry, and the models behind indicators must be made clear to all stakeholders. When making data public that could have a negative impact on their reputation, institutions such as hospitals, industry, pharmacies and other stakeholders will have little incentive to comply with reporting. Whether as a condition of reimbursement or motivated by public profiling, hospitals will need incentives to reveal data on resistant infection levels at their facilities, given the potential repercussions. Mandates must be implemented at both the national and international level, that requires data transparency, in order to ensure appropriate sampling.
In order to ensure transparency, there should be clear criteria of what characterizes an effective measure and a checklist of what concerns should be averted, such as disparate impact. We welcome the country self-reports that WHO has now collected as a steppingstone for engaging with country-level policymakers and as a sign of political commitment. Civil society views the country self-report as complementary to objective, independently verifiable indicators. The public posting of survey results would allow civil society to provide alternative data that might provide a quality check to the official version submitted by the Member State. It is admittedly difficult to know the direction of potential bias in self-reported country-level data. However, indicators from the M&E framework as well as civil society could complement effectively, if we plan ahead to construct the indicators for this purpose. Modeling also would help countries to set priorities among the interventions tackling AMR, and accordingly, the corresponding indicators. In such models, it will become obvious what variables are most sensitive to intervene upon and which ones will make the most leveraged difference. Models also lie behind some of the proposed indicators, e.g., “OG.4: Modelled estimates of mortality attributed to resistant infections e.g. for bloodstream and/or cerebrospinal fluid infections caused by carbapenem-resistant Enterobacteriaceae (CRE), vancomycin resistant enterococci (VRE), Clostridium difficile and MRSA, per 100,000 per year.” (Page 33) However, the model behind this indicator is unclear, when available, it should be made transparent to the public and policymakers, so that we can assess its value.
Despite the Tripartite’s request to select the most relevant or important indicators, discussions among ARC members could not lead to a consensus, without a clearer understanding of how indicators might be used in concert to effect follow-on change. We recommend that indicators be piloted to gauge feasibility in different settings and that all indicators be SMART: specific, measurable, attainable, realistic and timely, and most important, actionable by policymakers. The choice of indicators should take into consideration if it is most strategic to focus on the magnitude of the problem, or the progress made, and what framing will be most persuasive to policy makers. Other concerns noted from the assessment of indicators include:
Use of the Drug Resistance Index – As acknowledged by the Tripartite, there is a need “to consider further the potential use of an aggregate measure such as a Drug Resistance Index” (Page 16). Greater testing and evaluation must be done, as use of such an index, without assessing its potential shortcomings, could mask important underlying trends in individual drug resistance.
Reliance on global medians – Using global medians as a benchmark to signal overuse or underuse is problematic. Use of global median measures needs to account for diverse contexts in order to avoid signaling to a country that it must reduce the levels of antibiotics it uses, without distinguishing between settings where underuse and overuse are both problems.
Affordability Index – Additional efforts should be made to further develop “B.5: Affordability of key products – e.g. number of countries with price of a course of a specific, generic second line antibiotic is more than 7 days’ wages.” (Page 32), using the wages of the lowest paid government workers salary as a benchmark.
Lastly, there should be a measure of the political commitment of national governments to the NAP on AMR. This measure could aim to capture several dimensions of political commitment, including a country’s level of compliance with reporting, investment or recruitment of necessary financial and technical resources, and the implementation of follow-on plans when progress on AMR lags.
The AMR M&E should include progress on all the elements of GAP objective 5, not limited to those mentioned in 3.c/d in the draft document.
The proposed R&D indicators are currently aligned with the fifth strategic objective of the GAP “Increased R&D on new medicines, diagnostics, vaccines & other interventions related to priority pathogens.” (Page 18), and in line with the Global Framework for Development and Stewardship to combat AMR (Page 30), yet there is need for additional indicators.
It is important that R&D on AMR is not monitored only by extent of investments but also that it be in line with the principle of de-linkage, so price is divorced from research and development costs, as well as from sales and volumes in such a way to allow R&D to “facilitate equitable and affordable access to new medicines, diagnostic tools, vaccines and other results” as called for in the UN Political Declaration on AMR.[ii] For the principle of de-linkage to be met, both push and pull incentives are required, coupled with specific requirements for access and stewardship.
The tracking of funding flows for R&D is critically important, and while covered among the proposed indicators, it must be disaggregated to better understand the form investments are taking. Disaggregation should include the different type of investments being made, either as push or pull incentives, at what stage of the product development pipeline, and to what type of entity (i.e. university, research institutes, product development partnerships, industry disaggregated by type). For example, late stage market entry awards may do little to relieve the scientific bottleneck in the antibiotic R&D pipeline and may only serve to reinforce a traditional pharmaceutical R&D model.
Additional indicators should be included to capture the fairness of returns on publicly funded R&D. This measure might qualitatively or quantitatively capture requirements placed on public-supported grant programs, funding to universities, prizes for antibiotic R&D and complementary technologies such as vaccines and diagnostics. Models that should be avoided include expanding market exclusivity, as it runs counter to the principle of de-linkage, as well as transferable IP schemes, which risk burdening other patients with cross-subsidizing antibiotic innovation. The transparency of R&D inputs into antibiotic innovation should be an important goal of the M&E framework, as it is a key element of ensuring fair drug pricing and fair returns on public investments in R&D.
Ecumenical Pharmaceutical Network (Mirfin Mpundu)
Health Action International (Tim Reed)
Heath Care Without Harm (Anja Leetz)
South Center (Viviana Munoz and Mirza Alas)
Third World Network (Yoke Ling)
ReAct Africa Node (Mirfin Mpundu)
ReAct Asia Pacific (Sujith Chandy)
ReAct Europe (Anna Zorzet)
ReAct Latin America (Arturo Quizhpe)
ReAct North America/Strategic Policy Program (Anthony So)
[i] World Health Organization, Interagency Coordination Group (IACG) on Antimicrobial Resistance draft work plan, 2017. Available at: http://www.who.int/antimicrobial-resistance/interagency-coordination-group/online-consultation/en
[ii] United Nations General Assembly, Draft political declaration of the high-level meeting of the General Assembly on antimicrobial resistance, 2016. Available at: http://www.un.org/pga/71/wp-content/uploads/sites/40/2016/09/Draft-AMR-Declaration.pdf