Guidance objectives, structure and target audience

In immunization supply chains, data too often are collected but not used for action. This guide aims to help in-country managers and staff make sense of the large amount of data they collect and help them prioritize and present critical data in more actionable ways through the use of supply chain dashboards.

Step 1: Perform a situational analysis

Before designing a dashboard or assessing how dashboards can benefit a supply chain system, it is useful to take stock of the existing situation first:

1. What are the strong and the weak points of the supply chain? What problems need to be solved? Are there problems with availability, quality or efficiency?
2. What are the strong and the weak points of the existing supply chain management information systems? Assess practices relevant to collection, availability, reporting and use of supply chain data, and remember that meaningful dashboards can often be designed using data already available at different levels.

A situational analysis of the strengths, weaknesses, opportunities and threats (SWOT analysis) should address the information systems as well as the following four key functions in supply chain management:

• Distribution and transportation
• Forecasting and demand planning
• Stock management
• Cold chain management

Information for this analysis can be obtained through a review of the existing information system tools and national supply chain databases (if available), through semi-structured interviews with supply chain managers and other users at various levels of the health system, as well as through a desk review of existing assessments such as:

• Most recent Service Availability and Readiness Assessment (SARA)
• Most recent Effective Vaccine Management (EVM) assessment
• Most recent Extended Programme on Immunization (EPI) or New Vaccine Post-Introduction Evaluation (PIE) review
• Any other reports on immunization supply chain management in country

Step 2: Define objectives

Once you have better understood the strengths and weaknesses of the supply chain and the existing information system, it is important to prioritize key functions and clearly define the objectives and desired outcomes, that is, how using a dashboard will improve the performance of each function, the immunization supply chain and the overall EPI programme. One approach to prioritizing key functions would be to consider which would contribute most significantly to achieving such immunization programme goals as increased coverage and reduced inequity in access to vaccinations. Alternatively, decision makers may choose to focus the dashboard on what the SWOT analysis reveals to be the lowest performing functions.

Common examples of outcomes from using a dashboard include:

• Lower wastage caused by temperature exposure through better performing cold chain equipment and shorter storage cycles at lower levels
• Higher availability of vaccines at service delivery level through better visibility of stocks and corresponding improvement in stock management
• Fewer temperature alarms of cold chain equipment through better temperature monitoring and timely maintenance and repair of poorly functioning equipment

Note that the outcomes do not have to have specific, quantified targets at this stage; rather, a general indication of how the poor performance can be best addressed through the use of dashboards can suffice.

Step 3: Engage key stakeholders

Ensuring that key stakeholders are in agreement with the objectives and outcomes expected from using a dashboard is essential for its successful planning, implementation and use. Key stakeholders include end-users at the health facility or district offices and those at the central level who have a role to play in providing financial, technical, political or staff resources to the development and use of dashboards.

• Health workers
• Health facility and service point level staff
• Sub-national health programme managers (e.g., district health officers and local health board members)
• National Logistics Working Group (if applicable)
• Supply chain/logistics department
• Expanded Programme on Immunization programme
• Ministry of Health leadership
• The key stakeholders to involve depend on the context; they commonly include (but are not limited to):  
• National-level Ministry of Health programmes and departments
• Health information system department
• Central medical (or vaccine) store
• Sub-national Ministry of Health staff and managers
• Sub-national immunization supply chain focal points (e.g., sub-national cold chain and store managers)
• Health facility managers
• Technical partners from supply chain and immunization-focused international or local organizations present in the country (including United Nations agencies)

The first step in engaging with stakeholders is to obtain their political approval to implement a dashboard and validate the results of the previous steps. This may be done through face-to-face meetings during which the findings of the situational analysis of Step 1 and the objectives and desired outcomes of Step 2 are presented. Once approval has been obtained, key roles and responsibilities can be assigned to relevant stakeholders. These roles can be either unique or repeating. Unique roles are those that relate to the implementation and continued management of the dashboards; they mainly occur at only one, or the central, level, such as the roles of the lead coordinator and members of the technical working group and steering committee. Repeating roles are those required to continuously generate and act on the dashboards; they exist at every level of the health system. Once the dashboards have been fully implemented and their use is considered part of the health workers’ day-to-day duties, the unique role responsibilities decrease while the repeating roles remain the same or even increase.

Table 2 lays out some of the most common roles and responsibilities, and the corresponding personnel for each.

To ensure seamless coordination, it is important to identify and provide terms of reference for each member with a unique role. For those with repeating roles, it is enough at this stage to identify the specific job posts that will be affected by dashboards; later, in the implementation phase, detailed training will equip them to properly carry out their responsibilities.

Step 4: Define primary key indicators and data requirements

After stakeholder approval and validation of the objectives and desired outcomes has been obtained, it is necessary to select indicators, the critical data that will allow managers to use the dashboard to measure performance and take corrective action if necessary. While there are many possible indicators, of varying complexity, it is important to weigh the costs of collecting the data necessary to use them. Accordingly, selection of indicators must strike a balance between level of detail and the feasibility of implementation in terms of both cost and time.

This guidance presents seven primary key indicators (see Table 3) that, if used together, provide a comprehensive overview of the performance of the four key functions of supply chain management (forecasting and demand planning, stock management, distribution and transportation, and cold chain management). A dashboard need not employ all seven indicators: It may focus on one or more of the supply chain functions and use only the indicators pertinent to those areas. A country’s specific needs and context will determine whether additional indicators should be used and how dashboards can be tailored to various supply chain levels and users.

The calculations and visualizations relevant to each indicator apply differently on the dashboard at each supply chain level, as demonstrated by the case study from Pakistan (see below). A more detailed set of visualization options and potential corrective action to take for poorly performing indicators is presented in the Indicator Reference Sheets published in conjunction with this Guidance.

The dashboard will evolve over time as supply chains mature and users advance in their own capacity to digest and analyse data. Therefore, dashboards, and the underlying technology, must be agile enough to evolve. The immunization supply chain dashboards are still to be implemented in Pakistan, but from other commodity groups it is evident that managers and policymakers with access to dashboards make better decisions regarding stock availability, transportation, financing, storage and human resource capacity.

Some considerations for adapting the indicators to individual settings are listed below:

• Using tracer products or all products: Indicator performance can be measured for all vaccines and supplies managed by the immunization programme or through a subset of ’tracer‘ products. Similarly, all stores and facilities or a sample of them can provide data for calculating indicators.
• Data collection and reporting frequencies: The appropriate frequency for data collection, reporting and analysis depends on a number of contextual factors, such as the burden of collecting data, how often and how quickly corrective actions can be taken and the frequency of interpreting the dashboards.
• Target performance: The visualizations in the Indicator Reference Sheets occasionally state thresholds for good performance. These do not represent recommended thresholds; instead, countries are encouraged to set their own thresholds based on local immunization supply chain performance targets.

Once the primary key indicators have been chosen, the data requirements to visualize them can be documented. The Indicator Reference Sheets include the data requirements and data sources for all seven of the indicators listed in Table 3 above.

Step 5: Gap analysis

Having defined the data requirements for the chosen indicators, the country must ensure that the proper tools, records, and data collection and reporting processes exist to satisfy the requirements. Thus, it is necessary to return to the results of the situational analysis undertaken in Step 1 and re-examine the existing supply chain information systems.

In Nigeria, a separate – phone-based – reporting system was introduced in addition to the existing LMIS to provide data for the Stocked According to Plan indicator.

In the United Republic of Tanzania, the development of a new electronic Vaccine Information Management System incorporates data requirements for calculation of supply chain indicators.

Within these systems, both recording and reporting functions need to be carefully mapped out to help identify gaps and processes that can be put in place to acquire the necessary data. Once the gaps have been identified, the difficulty of addressing them with the available resources and any additional burden new procedures might place on health workers must be considered, as demonstrated in the following table:

 

 

 

 

Step 6: Implementation plan

The implementation plan details the activities, tasks, budget and training required to implement the dashboard. There can be two implementation phases, depending on a country’s course of action in response to the previous steps:

1. Implementation plan for a dashboard using the existing information systems
2. Implementation plan for new mechanisms (e.g., data collection and reporting processes) necessary to obtain missing data

In both phases, it is important to:

• People, process and technology: Ensure not only that the new tools and technologies are well developed, but that adequate training for the health personnel who will use them is provided for and that efficient processes to integrate them in the existing systems are in place.
• Build on the existing information systems: Make sure key information system personnel are involved, and build on the existing system in an integrated way that adds value and does not create parallel workflow. This may or may not involve personnel from the health ministry’s health management information system (HMIS), depending on whether EPI manages its data separately from the larger HMIS.  
• Start small, think big: When changes to the processes, tools and workflow are necessary, they need to be developed and piloted first. Because new systems can have heavy resource requirements, pilot them only at certain supply chain levels, and at a small sample of sites, before scaling them up. However, always plan the pilot with the feasibility of scaling up in consideration.
• Monitoring: It is always important to monitor or assess the impact of new systems. Implement proper monitoring not only of primary key indicators but also of process indicators to ensure that usage of new information systems (if applicable) and dashboard is tracked.

Because this guidance is primarily concerned with planning, implementing and using dashboards, more specific guidelines on how to plan for new data collection and reporting mechanisms is not covered here. For this, the information and communications (ICT) guidance ‘Planning an Information Systems Project: A toolkit for public health managers⁴ may be helpful.

Including dashboard implementation plans in overall EPI programme goals and strategic documents ensures that dashboard implementation remains on the policy agenda, which makes it easier to fully implement a long-term plan for optimized dashboard use as part of an overall supply chain information system improvement plan.

⁴Available at <www.path.org/publications/files/TS_opt_ict_toolkit.pdf>.

Implementing a dashboard often involves three phases: pilot, scale, sustain.

Step 1: Pilot

The pilot serves as a ‘test’ of the implementation processes and the dashboard tool undertaken on a small scale or within a limited geographic area. The lessons learned from a pilot can inform key stakeholders when the time comes to scale up the use of the dashboard.

Approaches to improve reporting rates
In Nigeria, local government areas (LGAs) sent health facilities reminders and follow-ups and reached a reporting rate of around 90 per cent.

In Karnataka State in India, a televised ‘bulletin board’ showing the performance of different health facilities improved the reporting rate.

In Mozambique, inconsistent reporting rate led to a shift in reporting responsibility from health facility to field coordinators using tablets for electronic data entry.

In the pilot phase, it is important to track:

1. Are the processes in place to generate and use the dashboards appropriately?
2. Are facilities or stores reporting in a timely and complete manner?
3. Are the health workers and supply chain professionals adequately performing their tasks?
4. Does the use of dashboards result in performance changes?

The lead coordinator and the dashboard steering committee should review each of the above four areas on a regular basis. Monitoring of these areas should inform changes to the implementation plan, both for the pilot and scaling up.

The specific key tasks in the pilot phase are:

1. Communicate the detailed workplan to all stakeholders. The lead coordinator ‘owns’ the workplan and is responsible for making sure its key milestones are achieved and for identifying potential problems that should be referred to the relevant manager or committee.
2. Conduct a baseline assessment to measure the pre-dashboard ’before‘ state, so that it can be compared with the ’after’ state. It is important to measure both the selected primary key indicators (e.g., time taken to respond to a stock-out, full stock availability, etc.) and process indicators related to adoption and use of the dashboards (e.g., reporting rates, percentage of facilities using dashboards, etc.).
3. Train staff in using and creating dashboards. Training should focus on the dashboard users (e.g., supervisors and managers), data collectors (e.g., health facility staff or warehouse managers) and data managers at each level. Tailoring training material to the context-specific workflow can improve the usefulness of the training (see How do dashboards support your immunization programme?).
4. Hold regular dashboard review meetings with the relevant committees or working groups. It is important to integrate dashboard reviews with the existing structures for management of the supply chain. The review can focus on supply chain performance progress and dashboard use, processes and design. The technical working group should spearhead this effort using key forums or points of entry for dashboard discussions such as: 
   • Regular (monthly or quarterly) Expanded Programme on Immunization (EPI) management meetings at national, sub-national or district levels
   • National logistics working groups⁵ or sub-national logistics working groups (such as state, provincial or district groups)
   • Regular (monthly or more frequent) health facility meetings

Visualizing data and providing feedback to the data collectors through a dashboard may itself increase the likelihood of reporting, as data collectors see the importance of the reporting and supervisors can focus support to the low reporting sites. It may be desirable to specifically visualize data completeness and/or timeliness on a dashboard, particularly if reporting rates are low or are being targeted for improvement.

⁵Alternate names include Vaccine Management Committee, Cold Chain Working Group, Supply Chain Technical Working Group, etc.

Step 2: Scale

Upon conclusion of the pilot, there should be a decision to scale it up, continue it or stop dashboard implementation. If the decision is to scale the dashboards, it is important to focus not only on their adoption and use but also on institutionalizing them, so that using them becomes a natural part of health workers’ responsibilities – ideally with Ministry of Health policies in place describing how dashboards are to be used by everyone working within the immunization supply chain.

Nigeria’s Vaccine Stock Performance Management dashboard: ‘Pilot’ to ‘Sustain’

Pilot: Kano and Lagos States were selected for the pilot stage. Weekly dashboards including three indicators were used over an 8–9 month period.

Scale: The scale-up stage took 4 months and used a training cascade – from national training of trainers to training of health workers in state stores and then LGA stores – to train staff in data reporting and using the dashboards.

Sustain: To reduce manual aggregation, there are efforts to integrate dashboards in the automated warehouse management system. This would reduce the time health workers spend providing data to the dashboards and make it possible to extend the dashboard from

The scale-up phase includes many of the same activities as the pilot, but extends them in terms of geography or of supply chain levels. For example, a dashboard piloted at all supply chain levels in a given district may be scaled to more districts and eventually to an entire country. Or dashboard use may begin at the national level and scale down to extra districts and health facilities.

The key tasks in the scale step are:

• Continue pilot activities: It is important to conduct baseline assessments, training and monitoring of reporting rates and performance in all new implementation sites, just as in the pilot phase. Dashboard users from the pilot phase can help to train the new users.
• Adapt or revise dashboards: Experience acquired during the pilot phase may prompt changes to the content or the processes surrounding the use of dashboards. Remember to communicate changes to all health workers affected. Additional training might be necessary if changes are extensive.
• Integration into management systems: Dashboards should not be viewed as a stand-alone tool but as part of the existing management systems and structures. Integrating dashboards into the existing supervision and management processes is especially critical at the scale-up phase, as the lack of integration leads to parallel workflow and an additional burden on health-care workers at the extended scale (district, provincial, state or national).

Scale can occur over a short or long time frame depending on the complexity of the change and its feasibility (e.g., having the right people, processes, technology and policies in place).

Step 3: Sustain

Once scale has been achieved, the processes surrounding the creation and use of dashboards have been normalized and dashboard use has become part of regular management and oversight procedures. The third step, sustain, focuses on monitoring both how well the dashboards are being utilized and their impact on the wider programme. Areas needing improvement should be identified and addressed continuously. In addition to some of the short-term solutions, long-term improvements to dashboard use and the overall information system can include addressing:

• Data quality and timeliness, assessed through data audits and improved through 1) revision of data collection tools and simplification of data collection, reporting and aggregation processes, and 2) supportive supervision and training to emphasize the importance of data quality.
• The burden of data collection and reporting at the health facility level, which will require identifying and limiting redundant processes through extensive revision of the immunization supply chain recording and reporting forms

 

Another important way to improve dashboard use in the long-term is to set up feedback loops to communicate performance progress to the health workers who collect the dashboard data. The more they understand how the data they provide are used at higher supply chain levels, the more they can appreciate the importance of their data collector role, which can spark their interest in reporting data of the highest quality. The result is to make data more reliable and actions at each supply chain level more appropriate to the actual situation at lower supply chain levels.

Where possible, automation can support and reduce the burden of data collection, reporting and analysis. It is important to note, however, that technological solutions do not solve dashboard reporting quality, rate and timeliness problems independently. A good principle to follow when considering implementation of technologies is the principle of ’Make it great, then automate‘. Once processes and analysis are being routinely followed in the paper-based system, then consider technological adaptations to automatically collect, update or analyse data.

When dashboards are implemented, they are likely to be used initially in challenging and imperfect systems, where their introduction alone will not bring about change. But the use of dashboards can drive a process of identifying problems and supporting change. The dashboard life cycle (see Figure 7) illustrates this process of continuous improvement in which change is reinforced and results in new supply chain management behaviours.

Because using dashboards for immunization supply chain management is a developing field, many new initiatives are taking place in a variety of countries.

In Karnataka State, the team assessed the number of transactions (indication of adoption and use of new LMIS), vaccine availability (indication of how the supply chain system performed) and log-in to the web dashboard (indication of use of data) When the assessment showed that the expected outcome had not been achieved, a bulletin board was introduced, which led to an increase in the transaction volume and to improved availability. Only the monitoring of the dashboard use highlighted the adaptation problems.

Those piloting such technologies are highly encouraged not only to refer to the publications and websites mentioned in Additional Resources for further reading, but also to share their experiences through knowledge-sharing platforms such as TechNet-21 (available online at www.technet-21.org). On this platform dedicated to the community of immunization professionals, a specific user group has been created to allow for posting discussions, sharing documents and collecting feedback from other colleagues regarding the use of supply chain dashboards.