GAVI Alliance Supply Chain Strategy > Using data for better supply chain management

Thank you Oz for your thoughtful reply. I agree and would like to respond to a couple of your questions. Performance indicators: Availability: Absence of stockouts is the best surrogate for disruption of immunization sessions that we can find. We tried and tested a direct disruption indicator in the Indonesia Sreamlining project in 2003 and found that disruption is perceived as management failure that noone wants to report - failing to meet community needs is seen as more serious than stockout. I dont buy the notion of 'planned' or 'tolerated' stockout because there should always be vaccine available for a child, not just for planned sessions. System cost: I dutifully include system cost in the montioring indicators but I'm skeptical that this can be recorded and reported as a routine; I would really like to hear from others because I think that system cost can be evaluated from time to time, but not monitored (too many different inputs, different sources, different responsiblities) Temperatures: I doubt that we can monitor the use of VVM-compromised vaccine; I dont think anyone would admit to that! Vaccine utilisation: It is simpler, clearer and better interpreted to orient all performance indicators the same direction - usually 'good' is a higher score. But vaccine utilization is a little different from the inverse of wastage because there are several measures of wastage and only one for utilization. Utilization is simpler to calculate and does not require extra recording. Wastage studies, evaluations are able to collect mutliple wastage indicators accurately, but routine monitoring rarely achieves compliance or accuracy. However, we need to debate one remaining issue with Vaccine Utilisation /Wastage: Utilization varies with population density and infrastructure which we cannot control as well as session size, cold chain etc. which we can control. So Utilisation comparisons between countries are not valid unless we can 'normalize' by a factor correcting for density and infrastructure. In the meantime, we can only compare time series data. Thank you! John

Many thanks John for outlining some of the steps that are needed in countries - though we must rely on governments to set the systems up - supported by GAVI Alliance. (Interesting side question of whether this is best done through supporting countries to do so, or requiring them to obtain GAVI support). Thanks also for suggesting some of the key performance indicators, and wanted to explore this with the community. For me, the challenge is how to measure each of the three key parameters: availability, quality, and cost. (note that i prefer these classical ways to describe performance, but I think they mean the same as your three.) The most challenging to measure is 'full system cost' - so what could be measured now; and must be measurable once the better systems are in place? (You mention the 'vaccine utilisation rate', which can be defined as the arithmetic inverse of wastage - and this is probably the only measure of cost that current systems might obtain - however, most if not all of the 'opened-vial' wastage of this 'cost' is not preventable; so should focus only be on unopened vial wastage?? For availability - the key questions are not stock-outs at stores, but lack of vaccine at an immunization session. Is it possible to measure this? And what are the proxies to use instead For quality, the suggestion of using storage temperature alarms is a good proxy - but what we really want to measure is not vaccine damage (this is an efficiency issue) but rather tiem-temperature damaged vaccines being given. Is it possible to measure? Looking forward to further thoughts form the community on what (and how) to measure performance. Please also note the new discussion started on the issue for data for management at: http://technet-21.org/forums/default-topic/3275-data-for-management-strategy-framework#p4682 Looking forward to more ideas and suggestions Oz

I agree with John's list of KPIs. I think process indicators are essential for monitoring the routine processes (reporting or ordering, order fulfillment, # or % of emergency orders), while outcome indicators (stockout rates, wastage, cold chain functionality, etc.) address the overall system performance and impact. Here's the justification for process indicators: Facility reporting rate (and timeliness) directly affects the quality of reporting on which the higher levels depend for decisionmaking, and on which resupply to the facilities as well as cold chain service depends. It also helps pinpoint performance gaps and enables targeted intervention as well as comparison (and recognition/reward) among facilities, local governments/districts, etc. Order fill rate monitors the responsiveness to reports/orders of the higher level suppliers (stores) to the needs of the facilities. The Measuring Supply Chain Performance guide published by USAID | DELIVER PROJECT is an excellent reference tool that includes four types of indicators by supply chain function: quality, time, financial, and productivity. They do not address cold chain equipment or VVM per se, but I would suggest: # and % of facilities with functional cold chain equipment (functioning and within temperature parameters) per reporting period # and % of facilities and stores "at plan" for cold chain storage capacity response time per service call for equipment failures % of on time maintenance visits (according to service level agreement) That isn't intended as a comprehensive list. As far as what GAVI does with this, it makes sense to focus on the higher level system performance (outcomes) for reporting to GAVI, while emphasizing to country programs the value of using process indicators to address gaps that typically hinder system performance and impact outcomes.

Chris made some great points about transactional systems versus indicator monitoring. I think that both are important for better operations, management, and strategic decision making, and that we will only get good indicator data if we get better transactional management systems in place. It is also very true that technology is only part of the solution and that nothing will be achieved if we don't think about technology, people and systems together. Thanks for linking to the CRDM methodology, which is an important tool that can empower country decision makers to gain control of system functionality. I also wanted to highlight the ICT Toolkit which builds on that methodology and which is now available at Technet. It aims to help public health managers to plan for the implementation of information and communications technology (ICT) in health information systems through a number of steps, one of which is the definition of requirements. Finally, the people and organizations who are active in LMIS and data for management comprise an ever growing ecosystem. How can we harness their creativity and resources to achieve better supply chains and eventually better health outcomes. I posted a new thread on the D4M strategy framework that tries to tease out what needs to be done to achieve a vision like the one projected by John in his post. Thanks for chipping in there as well!

I believe that GAVI can help in establishing information systems for Supply Chain management. The critical 6 activities would seem to be the training and i-equipping to:

To establish inventory of facilities and equipment used in distributing vaccines, following standard proposed to UNICEF/CCL Based on the inventory a)prepare 5-year plan for vaccine storage and transport equipment b)establish GSM/GPRS based updating of location and condition of bar-coded equipment and related job-card history of repairs To establish pre-planned schedule of vaccine deliveries and track their execution, timeliness and completeness via internet To progressively equip all primary and intermediate vaccine stores with web-enabled stock control applications Based on the transaction records of the stock control system, use rate of consumption at service points and outreach to forecast vaccine need Create routine system of reporting 3-5 performance indicators on supply system quality, efficiency and effectiveness (Proposed indicators: storage temperature alarms; vaccine utilization rate; freedom from stockouts; delay between request and equipment repair response; system cost per dose distributed)

These activities would help to achieve a supply system that: has sufficient 'cool-capacity' in storage and transport for the future has coherent and defensible plans on which to base requests for external/government funding maintains a regular system of deliveries that can be relied upon Accurately forecasts vaccine requirements Respnds rapidly to callls for repair services Supvervises the system on the basis of performance

"For KPIs, one can measure reporting rates and timeliness, order fill rates, lead time for resupply, losses, emergency orders, and other indicators of supply chain performance" Thanks Chris. And good to see alignment of views; and especially in relation to the primacy of the system and how people use people use the data, and tot see this as a software issue. I want to follow up on the issue of key performance indicators and it seems to be that your list of KPIs above are a mixture of process and performance indicators; and while I don't thing there is a problem with using a mixture for KPIs, I wanted to get some more inputs on what are the key aspects of performance for any supply chain, The 'six rights' can operationalized into three performance parameters: availability [adequate vaccines available for every session], quality [not time-temperature damaged vaccines administered], and cost [total cost of vaccines, equipment and operational costs to get vaccine from port to child] . So, what KPIs can monitor each of these parameters, given the challenge of measuring each?

Dear Joseph: Good questions indeed! I would first reply that logistics data is used not only for M&E purposes, but for logistics transactions, particularly determining how much of each vaccine or consumable to supply to each facility for routine immunization. So Logistics MIS data is very different from HMIS data. Data flows up the pipeline (as with HMIS), so that decisionmakers can make informed decisions on supplies flowing down the pipeline (the uniquely transactional element). This difference is essential when determining system requirements. For example DHIS is a widely used system for managing HMIS data, but so far has been unable to support the transactional requirements of supply systems--requistioning/order, order fulfillment, inventory management, etc. LMIS data can and should also be used for M&E purposes to monitor supply chain performance, and to validate (and be validated by) HMIS data on coverage. Now to answer your questions: 1. Key data requirements for logistics will vary, but at the most basic, the truly essential data items are consumption (dispensed or used, depending on the commmodity, or issued for use) and stock on hand. Optimally, losses and adjustments (from transfers, returns to inventory, etc.) should also be collected. For vaccines, other data might be included as well, related to cold chain monitoring (especially functionality of cold chain equipment) or wastage rates. Inventory data includes receipts, issues, stock on hand, batch/lot number, and expiry dates. For KPIs, one can measure reporting rates and timeliness, order fill rates, lead time for resupply, losses, emergency orders, and other indicators of supply chain performance. The benefit of using LMIS data to validate HMIS data is apparent when calculating or validating wastage rates. Wastage reported though HMIS can be independently verified by consumption and stock-on-hand data reported through the LMIS--the difference between reported doses administered (HMIS) and vials issued to vaccinators (LMIS), adjusting for remaining stock on hand, provides a basis for validating reported wastages, especially when aggregated at the district, regional, or national level. 2. There are a number of organizations with the necessary expertise to help programs determine logistics data requirements. I can only speak on behalf of JSI, which has been doing this work in many countries through a variety of projects and funders, and has gained recent experience integrating vaccines and related supplies into other health commodity LMIS and to a limited degree into integrated supply chains. Others might include AMP, BioForce, MSH, i+solutions, for example. 3. There are standard tools for assessing supply chains or elements (such as LMIS) within supply chains to assess current performance, identify gaps, and develop interventions (PSM Toolbox is a good source, as is the USAID | DELIVER PROJECT). LMIS assessments require a mapping of existing tools and systems, mapping data and resupply processes, mapping and engaging stakeholders, and identifying/mobilizing leadership and resources in the country. Ultimately, the assessment must make a strong "business case" for investing in new systems, and inform a strategic planning process based on a common vision of the end-state: data visibility to ensure availability of vaccines and related supplies, and efficient supply chains that constrain costs (to the degree possible) and manage risks. This is key to building the stakeholder commitment to make change happen. For change management, this requires a communication and training strategy at every step--let poeple know change is coming, how they (or their work) will benefit, what their roles will be, when they will be trained, how they will be supported, how performance will be monitored. There must be active and committed champions at every level who are engaged in decision-making about the new system, and who can be change agents among their peers. There must be user support resources for post-roll-out support and mentoring, troubleshooting, bug fixing, etc. The change management component is a long-term investment and must be factored into the total cost of ownership of any new MIS solution or IT system. And the entire system most not remain static--systems must be put in place to enable continuous improvement and evolution to happen based on changing circumstances--new vaccines, new approaches, new facilities, etc. Sorry for the long-winded answer, but this is really only scratching the surface of a process that, to succeed, requires commitment and resources over the long term. Chris

One way for GAVI to help countries address the need for improved data visibility is to make this topic a specific area in GAVI grant applications. Many country systems for vaccines or other health commodities rely on paper or emailed Excel files to report logistics data and to requisition or order new stock, but these systems are burdensome and prone to user and system error. But program managers and staff can be complacent unless prompted to think anew about better, more functional systems. They must also be guided and supported on the difficult task of identifying their data requirements, optimizing (or even reengineering) their "business" processes, and identifying the best technology solution to meet their new requirements. Having gone through this process in a number of countries, I've observed the tendency to focus first on a specific software solution--whether commercial off-the-shelf (SAP, Oracle, Sage) or one provided by a well-meaning development partner--rather than on country-specific functional requirements. A focus on requirement first, combined with a metholodical reengineering and streamlining supply chain processes, is an essential step that must be taken before any software solution can be deemed appropriate. Fortunately, not every country needs to start this process from scratch. Countries have many unique needs within their logistics systems, but most systems share many common processes across supply chain functions (demand planning, procurement, storage, requisition, order fulfillment, etc.). The Common Requirements for Logistics Management Information Systems, published by PATH and used by JSI in a number of countries, enables country programs to go through the requirements and reengineering process without having to reinvent the wheel. The outcome is a set of country-specific or program-specific requirements against which specific software solutions can be judged. The other point to make is that technology is only part of the answer. The process of change management--adopting and sustaining new technology--is essential, and is much more difficult to accomplish. People must accommodate new processes, new ways of doing their work, and much greater reliance on and trust in systems that they may not fully understand. Resistance to change, particularly at high levels, should not be underestimated. Data visibility offers many benefits, but it also highlights failures, poor performance, waste that, with current systems, remain opaque or can be blamed on other factors.