Indicator Reference Sheets

In 2015, the Data for Immunization Supply Chain (DISC) indicators were developed and approved by the Gavi Alliance Partners to be used to monitor in-country immunization supply chains. The indicators are intended to be implemented at each level of the supply chain, so all managers can use them to manage the immunization supply chain. The indicators are selected, so they collectively provide an overview of the performance of the essential elements of the immunization supply chain. Likewise, the indicators are constructed to require simple data points to allow their use in settings with different degrees of data availability.

The DISC indicators are:

Click on each indicator to get more information including indicator definition, calculation and visualization examples.

Using one or all of the DISC indicators is the choice of each country depending on the availability of data and the capacity of the staff to use the information the indicators provide. The DISC indicators are only suggestions and do not need to replace already existing country indicators if they already cover the essential elements of the immunization supply chain.

For countries with advanced logistics management information systems (LMIS) and good data available, the indicators can be upgraded and more sophisticated. For example, the Functional Status of Cold Chain Equipment indicator is a simple version of uptime or downtime indicators. If the context allows, implementing the more sophisticated indicators will provide more information to the managers.

Equally important to selecting and implementing indicators to monitor performance is the use of the indicators to manage the immunization supply chain, thereby improving the services to the population. You can read more about visualizing and using data in the country cases.

Closed Vial Wastage

The indicator is used to measure potential avoidable wastage during transportation and storage. Wastage is related to the performance of vaccine ordering, distribution and store management. It can indicate excessive ordering practices that are not well-aligned to actual consumption rates, vaccine exposure to heat or freezing temperatures, breakage and mishandling of inventory.

This indicator can help answer questions such as:

Is wastage similar between facilities and between districts?
Do the quantities of vaccine ordered at particular facilities routinely exceed actual usage?
How many extra vaccines should be procured beyond those estimated to be administered?
What is the approximate financial value of closed vial wasted vaccine?
Is targeted reinforcement of standard operating procedures and vaccine management principles needed?

Expand this sheet Close this sheet

Closed Vial Wastage

Definition

Percentage of the total number of closed vial vaccine doses managed by a store or health facility during a particular period that are spoiled because of expiry, heat exposure, freezing, breakage, loss of the accompanying diluent or discard of unopened vials at the end of an outreach session. Wastage at the point of administration, because of incomplete use of the contents of a multi-dose vial, is referred to as open vial wastage and is not included in closed vial wastage.

Closed Vial Wastage

Performance objective

Availability
Potency
Efficiency

Closed Vial Wastage

Domain

Stock management

Closed Vial Wastage

Full indicator name(s)

Average closed vial wastage rate
Closed vial wastage rate per facility
Closed vial wastage rate per district/administrative level

Closed Vial Wastage

Dashboard use level

This indicator is recommended in dashboards used by sub-national and national managers and by store managers at all levels.

Closed Vial Wastage

Preconditions

A system for recording closed vial wastage, optionally with reason codes, needs to be in place.

Closed Vial Wastage

System design

Relevant in all types of logistics systems.

Closed Vial Wastage

Data needed

Number of discarded (wasted) doses reported by vaccine and preferably by reason code
Number of doses under management during a certain period, defined as the starting balance plus all of the doses received during that period

Closed Vial Wastage

Data sources

Vaccine stock ledgers/cards
Vaccine orders
Batch management to track vaccine vial monitor (VVM) status and expiry dates
Logistics management information system (LMIS)
Wastage reporting tools

Closed Vial Wastage

Calculation

Doses under management is defined as the opening balance plus all doses that were received during the period. Issued doses should not be subtracted.

Closed vial wastage should include vials wasted due to:

Expiry, which may indicate ordering practices that are not aligned to actual consumption rates, failure to respect first expiry first out (FEFO) policies, a supply design that moves too slowly (i.e., it takes too long for a vaccine to go through the chain to the point of administration) or poor organization in a vaccine store such that an older lot or batch can be overlooked.
VVM status 3 or 4 (at or beyond the discard point) before the vaccine’s expiry date has been reached, which may indicate poor cold chain quality or breaches in the cold chain.
Freezing, which is an indication of poorly functioning cold chain equipment or poor adherence to standard operating procedures during storage or transportation.
Breakage, either of the vials or accompanying diluent.

Inclusion of reason codes in reporting of closed vial wastage allows additional precision and more thorough investigation of root causes.

Example
In a regional store, 500 doses of pentavalent vaccines expired during the year and 240 doses were wasted due to VVM status 3 or 4, bringing the total for the period to 740 doses.

If the beginning balance of pentavalent vaccines in that same store was 5,000 doses, and four shipments of 5,000 doses were received during the year, then the total number of doses under management during the year was 25,000 doses (5,000 + (4 x 5,000).

When calculating by reason code, the overall closed vial wastage is divided into:

Closed Vial Wastage

Visualization and interpretation

The performance of this indicator can be visualized in a table that includes the number of doses and the percentage of doses that were wasted. Adding a target for closed vial wastage makes it easier to identify where actions are needed.

Closed vial wastage can also be visualized in a line graph to show the performance over time for different districts in a country.

The graph shows that districts 1, 3 and 6 have higher closed vial wastage throughout the year, while the other districts perform within the target range.

Closed Vial Wastage

Potential corrective actions

Perform root cause analysis to identify the reasons for closed vial wastage and identify areas for improvement based on the reason for wastage
Implement improvement activities
Develop or review relevant standard operating procedures for store and stock management

Closed Vial Wastage

Related indicators

Temperature Alarm Rate
Stocked According To Plan
Open Vial Wastage

Forecasted Demand Ratio

Used to validate and improve forecasting practices and assumptions (e.g., target population, coverage, wastage) in order to increase forecasting accuracy.

The indicator helps to answer questions such as:

Is consumption in a health facility, administrative unit or country as expected?
Is there a need to plan for additional stock to avoid stock-outs?
Is closed vial wastage likely due to lower usage than expected?
Is there a need to review the forecasting assumptions (e.g., target population, coverage)?
Is there a need to revise minimum and maximum stock levels?

Expand this sheet Close this sheet

Forecasted Demand Ratio

Definition

Ratio of actual consumption of a given product during a particular period compared to the consumption forecasted for the same period. Consumption includes administered and wasted doses.

Forecasted Demand Ratio

Performance objective

Availability

Forecasted Demand Ratio

Domain

Demand planning

Forecasted Demand Ratio

Full indicator name(s)

Health facility forecasted demand ratio
Average forecasted demand ratio for sub-national level
% of health facilities with forecasted demand ratio in a set interval

Forecasted Demand Ratio

Dashboard use level

This indicator is recommended in dashboards used by sub-national and national managers.

Forecasted Demand Ratio

Preconditions

Consumption (i.e., administered and wasted doses) data is necessary to calculate the indicator, so a system to collect actual consumption data is necessary.

Forecasted Demand Ratio

System design

Relevant in all types of supply chain systems.

Forecasted Demand Ratio

Data needed

Forecasted demand/usage by product
Actual consumption by product (opening balance + receipts − closing balance of product)

Forecasted Demand Ratio

Data sources

Logistics management information system (LMIS)
Monthly immunization reports
Micro plans
Stock ledgers/cards

Forecasted Demand Ratio

Calculation

Forecasted demand ratio = (doses consumed per product in a period/doses forecasted per product for the same period)

It is important that the doses consumed and the doses forecasted apply to the same period. The longer the period, the more accurate the forecasted demand ratio. A rolling year, half year or quarter are recommended, but the length of the period might depend on the reliable data available and the staff’s ability to calculate indicator performance for a long period.

Interpreting the ratio:

Forecasted demand ratio below 1: actual consumption (through administration and wastage) was less than the forecasted consumption for a given period.
Forecasted demand ratio above 1: actual consumption (through administration and wastage) was more than the forecasted consumption for a given period.
A forecasted demand ratio close to 1 implies that the forecasted consumption matched well with actual vaccine consumption.

Average forecasted demand ratio = (∑ health facility forecasted demand ratios)/(total # health facilities)

The indicator can also be expressed as the percentage of facilities with a forecasted demand ratio meeting certain criteria (for example, within the range of 0.7 to 1.3).

Example

In a health facility, the quarterly forecasted usage of yellow fever vaccine in 10-dose vials was 45 vials, whereas the actual consumption of this vaccine in the same quarter was 35 vials.

Forecasted demand ratio for yellow fever vaccine = 350/450 = 0.78

The forecasted demand ratio shows that the health facility’s actual consumption was lower than forecasted (forecasted demand ratio < 1).

In another health facility, the quarterly forecasted usage of yellow fever vaccine in 10-dose vials was 40 vials, and the actual consumption of this vaccine in the same quarter was 45 vials.

Forecasted demand ratio for yellow fever vaccine = 450/400 = 1.13
For this health facility, actual consumption was higher than the expected consumption, and vials from the buffer stock had to be used (forecasted demand ratio > 1).

District A is preparing its district report, all the facilities in the district having reported their forecasted demand ratio for the past quarter.

Average forecasted demand ratio (district A) = (0.78 + 1.13 + 1.50 + 1.25 + 0.85 + 0.93 + 0.98)/7 = 1.06

The average forecasted demand ratio shows that the overall district consumption is close to the consumption forecasted.

Another way to report the aggregated forecasted demand ratio is to calculate the percentage of health facilities with usage within set limits. In this example, a +/− 20% ratio is used. This method of calculation more clearly shows how many health facilities are consuming more or less than expected.

% of health facilities with forecasted demand ratio between 0.8 and 1.2 = 4/7 x 100 = 57%
57% of the health facilities in District A have consumption within the set target interval. The remaining facilities have either higher or lower usage than expected.

Forecasted Demand Ratio

Visualization and interpretation

The forecasted demand ratio can be visualized in a bar chart. Values above 1 indicate consumption above forecasted demand quantities, while a ratio below 1 indicates lower than expected consumption. Target lines can be inserted to make it easy to identify the health facilities where actual usage differs from forecasted usage.

A line graph can be used to show the forecasted demand ratio over time. This is helpful to give national or sub-national levels an overview of health facility or district performance.

The line graph shows that the forecasted demand ratio is fairly constant in most of the districts, but in District 1, there is a variation over time that could be investigated further.

A spatial display is another way to give an overview of district performance. In this example, the colours indicate whether more or less than 80% of a district’s health facilities have a forecasted demand ratio between 0.8 and 1.2. The targets used are illustrative and can be adapted to the context.

Red = District with <80% of health facilities with forecasted demand ratio between 0.8 and 1.2
Green = District with >80% of health facilities with forecasted demand ratio between 0.8 and 1.2

Forecasted Demand Ratio

Potential corrective actions

Verify the actual usage with health facilities
Review the forecasting methodology and perform a root cause analysis to identify reasons for forecasted demand ratios beyond the established tolerance level (e.g., stock-out can lead to a forecasted demand ratio < 1). Root causes could be: inaccurate assumptions (target population, coverage and wastage), inaccurate on-time and in-full deliveries, higher wastage than expected.
Revise ordering policies and practices when the forecasted demand ratio is consistently outside of the tolerance level or there is a large imbalance
Revise minimum and maximum stock levels when forecasted demand ratio is consistently too high or too low

Forecasted Demand Ratio

Related indicators

Stocked According to Plan
Full Stock Availability
Closed Vial Wastage
On-Time and In-Full Delivery

Full Stock Availability

Measures the availability of immunization products. Availability of vaccines and immunization supplies is important to reach immunization programme targets.

Are certain facilities frequently at risk of stock-outs?
What is the full availability percentage by district or region?
Does low availability in the national or resupply store affect availability at lower levels?
Is full availability lower than expected in certain health facilities or regions?

Expand this sheet Close this sheet

Full Stock Availability

Definition

Percentage of storage points with full availability of all or a selected set of tracer vaccines and immunization supplies over a resupply period. Full availability is defined as no stock-out in the store or health facility at any point during the time period.

Full Stock Availability

Performance objective

Availability

Full Stock Availability

Domain

Stock management

Full Stock Availability

Full indicator name(s)

% of health facilities with full availability
% of districts with full availability
% of districts with at least x% of facilities with full stock availability

Full Stock Availability

Dashboard use level

This indicator is recommended in dashboards used by sub-national and national managers.

Full Stock Availability

Preconditions

This indicator can be implemented in any context, as it requires only observation of zero stock balance during the resupply period.

Full Stock Availability

System design

Relevant in all types of logistics systems.

Full Stock Availability

Data needed

Product stock-outs in stores and health facilities
OR: closing balances at the end of the resupply period in stores and health facilities

Full Stock Availability

Data sources

Stock cards/ledgers
Physical inventory/physical stock counts
Stock-out reports from health facilities
Logistics management information system (LMIS)

Full Stock Availability

Data collection method

Where necessary, full availability can be determined for a basket of tracer indicator products representing the availability of immunization supplies.

Full Stock Availability

Calculation

Full stock availability = resupply periods without stock-out of any (tracer) vaccine or immunization supplies

At sub-national and national level, the indicator is aggregated as % of health facilities or % of districts with full stock availability. The calculation for a sub-national region is:

% health facilities with full stock availability = (# health facilities with full availability of all (tracer) immunization products)/(total number of health facilities in sub-national region) x 100
Alternatively, for the national level, the aggregation can be based on the percentages of health facilities in a district exceeding a set threshold.

Districts with full availability of all (tracer) immunization products in more than x% of health facilities = (# districts with more than x% health facilities with full availability of all (tracer) immunization products in the last resupply period)/total # districts) x 100

The percentage of health facilities in the above calculation is set by the country to reflect the expected standards. When reporting the value of the indicator, the threshold value must be included.

Example
The table below shows health facility A’s report to the district on stock availability for the country’s tracer immunization products in the second quarter (Q2). Deliveries to the facility are monthly.

According to the table, there was full availability of all tracer immunization supplies in health facility A in April, but in both of the other months, at least one vaccine was not fully available. Therefore, health facility A had full stock availability only in April.

When the full stock availability percentages for each district are received at the national level, the national stock availability can be calculated as a national average or as a percentage of districts above a set percentage of health facilities with full availability.

National full stock availability = # health facilities with full stock availability/total # health facilities x 100 = 146/191 x 100 = 76%

The country has set 80% as the defined threshold for health facilities with full stock availability.

% districts with at least 80% of health facilities with full stock availability = # districts with >80% health facilities with full stock availability/total # districts) x 100 = 6/11 x 100 = 55%

Full Stock Availability

Visualization and interpretation

A colour-coded table can be used to quickly identify district performance. The threshold for green and red performance has to be set according to the context and the availability. Here, 80% was used as the threshold.

Another way to visually represent full stock availability percentages by district is through colour-coded spatial analysis. In the visualization of a region below, 80% was again used as the performance threshold. In the red districts, fewer than 80% of health facilities (HF) have full availability of a set of tracer vaccines and immunization supplies, while in the green district more than 80% have full availability.

Red = Full stock availability < 80% of HF
Green = Full stock availability > 80% of HF

Full stock availability can also be shown in a line graph plotting performance over time.

Full Stock Availability

Potential corrective actions

Verify the full availability of products in the past resupply period
Perform root cause analysis to identify the reason for low stock availability, including inventory management, reorder policies (push or pull), distribution plans, national stock availability and distribution performance
Review emergency resupply policies if there is a historical pattern of low stock availability
Review supply pipelines and planned orders for stores

Full Stock Availability

Related indicators

Stocked According To Plan
Average Duration of Stock-Outs
Average Response Time to Resolve Stock-Out
Month of Stock
Open Vial Wastage

Functional Status of Cold Chain Equipment

Measures operational cold chain equipment to identify where maintenance is needed for maintaining vaccine quality. Used for operational purposes, such as updating the maintenance plan, and for strategic purposes, such as to plan for replacement.

Over time, the trend in the proportion of functional equipment can be used to measure performance of in-house or contracted maintenance and repair services. If the proportion of functional equipment is disaggregated by reason for the non-function or by equipment type, the indicator can also be used to assess the performance of particular types or models of CCE in the field.

Note that functional status of CCE does not include a provision regarding the temperature maintained by the equipment; other indicators (such as Temperature Alarm Rate) must be used to fully understand the cold chain management system.

The following questions can be answered by monitoring this indicator:

What investment in new CCE is needed for the next few years?
Which CCE is in need of repair or maintenance and where is it located?
In case of delivery of additional CCE, where is it most needed?
Do particular models or types of CCE perform more reliably or have a longer lifespan than others?

Expand this sheet Close this sheet

Functional Status of Cold Chain Equipment

Definition

Cold chain equipment functioning compares the proportion of cold chain equipment (CCE) operable for storing vaccines with the overall number of commissioned CCE devices in a particular area. CCE is defined as all refrigerators, freezers, passive storage devices, and walk-in cold rooms and freezer rooms designated for storing vaccines. CCE functioning can be measured at a point in time or over a particular period of time.

Functional Status of Cold Chain Equipment

Performance objective

Potency

Functional Status of Cold Chain Equipment

Domain

Cold chain management

Functional Status of Cold Chain Equipment

Full indicator name(s)

% of functional CCE
% of health facilities or % of districts meeting a threshold for functional CCE (e.g., % of districts with at least 90% functional equipment

Functional Status of Cold Chain Equipment

Dashboard use level

This indicator is recommended in dashboards used by sub-national and national managers and all store managers.

Functional Status of Cold Chain Equipment

Preconditions

The indicator requires an updated cold chain equipment inventory, a mechanism to ascertain whether equipment is functioning and a system to transmit the information to the level where cold chain equipment planning is undertaken. The transmission mechanism can be paper-based, electronic or communication-based (e.g., by telephone).

Functional Status of Cold Chain Equipment

System design

Relevant in all types of logistics systems.

Functional Status of Cold Chain Equipment

Data needed

Primary reason for not functioning or not in use: needs spare parts/no finance/no fuel/surplus/dead/not applicable
Number of CCE devices designated for storing vaccines in a particular geographical area
Functional status of each CCE: functioning/awaiting repair/unserviceable
Optional additional data: temperature of CCE

Note: Precise definitions of the functional status and reasons for not functioning need to be standardized to allow comparison. For instance, CCE operating outside the normal range of temperature may be considered awaiting repair. Power sources, such as generators for backup power for walk-in cold facilities, may also be included in this indicator, as appropriate. Multiple reason codes may also be applied.

Functional Status of Cold Chain Equipment

Data sources

Maintenance worksheet
Cold chain equipment inventories by location; for example, WHO Excel-based tools such as Cold Chain Equipment Inventory (CCEI), Cold Chain Equipment Manager (CCEM)
On-site assessment of equipment functioning
CCE distribution plan

Functional Status of Cold Chain Equipment

Calculation

% CCE functioning = (# functioning CCE devices)/(total # CCE devices designated for use in reporting facilities) x 100

The indicator can be calculated either at a point in time or over a period of time. When calculated over a period of time, % CCE functioning needs to take into account how long the non-functional periods were:

% CCE functioning = number of functional CCE unit-days/total number of CCE unit-days in a given reporting period x 100,
where CCE unit-days are the total number of days in the reporting period multiplied by the number of CCE devices.

Both the numerator and the denominator should be collected from the same geographic area, and decommissioned equipment should not be counted in either the numerator or denominator. Functionality of CCE is broadly meant to mean that the device is operable at a particular point in time for storing vaccine.

Disaggregation of both the numerator and denominator by location and by type, manufacturer, model, energy source, PQS (performance, quality and safety) code or year of installation can add value in investigating root causes of CCE failures, in targeting maintenance and replacement, and in performance monitoring of equipment and of maintenance systems.

Examples
Consider a district with 50 facilities. The most recent CCE inventory indicates that the following equipment is available in the district, and a recent facility survey found the following number and percentage of devices functional:

Additionally, the district had 3 decommissioned refrigerators that were not counted in the above table.

Overall:
(42 functional devices)/(55 total devices) x 100 = 76% functioning

Consider a country with five regions. Recent data from a CCE inventory and a facility survey found the following equipment functional:

Overall:
(766 functional devices)/(835 total devices) x 100 = 92% functioning
(3 regions with >90% CCE functioning)/(5 total regions) x 100 = 60% of regions with >90% CCE functioning

Functional Status of Cold Chain Equipment

Visualization and interpretation

At the sub-national level, data from all facilities or an aggregate across facilities in a relevant area can be presented in a table format. A pie chart or sorted table can display the proportion of equipment that is functional, and managers can then use this information to monitor performance of maintenance systems and of particular types or models of CCE.

The performance of particular types of CCE in a district can also be visualized in a table or a pie chart.

For more information, and to assist in root cause analysis, the reasons for non-function and non-use can be illustrated in a stacked bar chart.

A line graph can be used to track the trend in the proportion of functional equipment over time.

A map can also be used to identify the need for repair or replacement of equipment geographically. Pins or markers can be used to identify facilities with a low proportion of functioning CCE.

At the national level, data are further aggregated across sub-national administrative regions or as a single estimate of functionality nationwide. Pie charts and line graphs are also useful at the national level.

Functional Status of Cold Chain Equipment

Potential corrective actions

Verify that equipment is not functioning
Determine the root cause of equipment dysfunction; solicit repair or replacement of non-functional equipment
Ensure that contingency plans are in place for all facilities, so that vaccines can be safely stored or transported elsewhere when one or more devices are non-functional
Perform routine maintenance of all CCE to prevent future breakdown
Use equipment status (including reasons) to inform future procurement decisions
Reallocate functional CCE equitably, as appropriate
The indicator can also be used in combination with other inputs, such as a cold chain inventory, to estimate the total volume of cold chain space available and is useful in assessing whether there is adequate functional cold chain capacity to meet needs for routine immunization, campaigns and new vaccine introductions

Functional Status of Cold Chain Equipment

Related indicators

Temperature Alarm Rate
Temperature in Range
Number of Maintenance Visits, Requests and Repairs
Cold Chain Equipment Uptime
Cold Chain Capacity Utilization
Mean Time to Repair Cold Chain Equipment
Mean Time to Implement Corrective Action
Mean Time between Refrigerator Failure

On-Time and In-Full Delivery

Used to ensure the store has the ability to meet the needs of lower-level stores, as well as the timeliness and reliability of order deliveries. The indicator can be used to monitor incoming shipments and performance of in-country distribution by the national store or outsourced distributor.

Including the indicator in a dashboard can facilitate store management improvements: increased reliability, consistency (client receives product needed each resupply period) and efficiency (reduction in emergency orders).

Note that OTIF delivery does not consider damage to products during distribution (e.g., broken vials, VVM stage 3 or 4). Other indicators (such as Closed Vial Wastage or Temperature Alarm Rate) should be used to identify such issues.

The following questions can be answered by monitoring this indicator:

Are orders correctly distributed in terms of products and quantities?
If warehousing and/or delivery services are outsourced, have the third-party logistics providers achieved their agreed-upon/contractual service levels in terms of timeliness, accuracy and fulfilment?
Are deliveries received during the expected time period?
Are orders correctly picked and packed in terms of products and quantities?
Have global procurement service agents and freight forwarders delivered products in-full and on-time?

Expand this sheet Close this sheet

On-Time and In-Full Delivery

Definition

Percentage of deliveries delivered on-time and in-full (OTIF), with OTIF defined as:

On time: Order is delivered when expected (e.g., on a specific date or within a specified time range)
Order fulfilled: Store can fulfil the complete order (i.e., provide all products and quantities requested)
Accurate: The correct products are delivered in the correct quantities (i.e., delivered products and quantities match the delivery note)

On-Time and In-Full Delivery

Performance objective

Efficiency
Availability

On-Time and In-Full Delivery

Domain

Distribution
Stock management

On-Time and In-Full Delivery

Full indicator name(s)

% of orders delivered on-time and in-full (OTIF)

On-Time and In-Full Delivery

Dashboard use level

This indicator is recommended in dashboards used by national and store managers at all levels.

On-Time and In-Full Delivery

Preconditions

This indicator is relevant in supply chains where:

Client knows the amount and/or expected amount
Delivery schedule is in place and date dispatched/received is captured
Stores deliver supplies to lower level stores or facilities (outbound delivery)

On-Time and In-Full Delivery

System design

The indicator is relevant for these supply chain systems:

Push system with fixed quantities
Pull system with delivery

On-Time and In-Full Delivery

Data needed

Scheduled delivery date or delivery range
Order requested by product and quantity
Order picked and dispatched by product and quantity
Products, quantities and time of receipt for dispatched orders by order

On-Time and In-Full Delivery

Data sources

Vaccine arrival report
Proof of delivery
Order delivery note
Submitted requisition/order
Delivery schedule
Advanced shipment notification

On-Time and In-Full Delivery

Data collection method

Data for this indicator is to be collected and compiled by the store responsible for fulfilling the orders. If the data collection systems are manual, sampling or sentinel sites can be used to collect data for calculation of OTIF. If the sample is large enough, this method will give a good picture of the actual performance of the system.

On-Time and In-Full Delivery

Calculation

% of orders delivered on-time and in-full = (# orders delivered OTIF/total # orders delivered) x 100

For stores that are not able to measure all three processes, an intermediate indicator (such as % of on-time deliveries) can be used.

Examples
Consider a regional store that picks, packs and delivers to four district stores once a month. The date of delivery, scheduled delivery date and data on quantities ordered, dispatched and received were collected from the relevant data sources and compared to identify if orders were delivered on-time and in-full.

% deliveries OTIF = (# orders OTIF/total # orders delivered) x 100 = (2 /4) x 100 = 50%

This regional store makes half of its deliveries on-time and in-full.

The National Logistics Working Group (NLWG) wants to discuss on-time and in-full deliveries and needs to aggregate regional and national store performances for the past quarter.
The following table shows the number of each regional store’s total deliveries that were delivered on-time and in-full during the months of the first quarter.

On-time and in-full delivery by the different stores during the first quarter varies from less than 50% to more than 91%.

On-Time and In-Full Delivery

Visualization and interpretation

The National Logistics Working Group (NLWG) is reviewing the performance of the three regional stores for the past quarter. The NLWG had established three initial thresholds to visualize and manage national and regional store performances.

OTIF <60% (critical, red)
OTIF >60% and <80% (priority, yellow)
OTIF >80% (normal, green)

Regional store C has been asked by the NLWG to present its March and quarter disaggregated OTIF performance data on what contributed to the poor OTIF.

On-Time and In-Full Delivery

Potential corrective actions

Assess system or policy changes (e.g., outsourcing or changing distribution system)
Improve forecasting and procurement procedures to ensure adequate stock at supplying stores
If services are outsourced, review past performance with warehouse and distribution service providers and agree on improvement actions
Improve or define standard operating procedures where needed
Revise demand plan to ensure adequate stock at supplying store
Adjust delivery schedule dates according to the actual capacity of the transportation services, if necessary
Negotiate with procurement service agents and freight forwarders on in-bound shipments to the country
Review and/or revise inventory policies including buffer stock and minimum and maximum levels for stores

On-Time and In-Full Delivery

Related indicators

% of Deliveries with Damaged Items
On-Time Dispatches
In-Full Arrival
On-Time Arrival
In-Full Dispatches
Order Accuracy
Vendor On-Time Delivery

Stocked According To Plan

Used to monitor and manage immunization products and as a warning to avoid stock-outs or wastage. Diversions from the planned stock levels can signal risk of stock-outs (if significantly below the minimum level) or closed vial wastage (if significantly above the maximum level). For stores, the indicator performance provides information on the ability of the store to dispatch the products and quantities needed by the health facilities.

Are the demand methodology and assumptions adequate?
Is there a risk of overstock and expiry?
The following questions can be answered by monitoring this indicator:
Is there a risk of stock-outs?
Will the supplied quantities be enough until next delivery?
Are the inventory policies and practices adequate?

Expand this sheet Close this sheet

Stocked According To Plan

Definition

This indicator measures the percentage of health facilities or stores maintaining appropriate (as defined by local policies) levels of vaccine and immunization product stock during a particular time frame, as compared to the overall number of facilities in the area. Stocked according to plan (SATP) is defined as stock levels between set minimum and maximum levels.

Stocked According To Plan

Performance objective

Availability
Efficiency

Stocked According To Plan

Domain

Stock management

Stocked According To Plan

Full indicator name(s)

% of districts with x% of facilities stocked according to plan
% of health facilities stocked according to plan
% of stores stocked according to plan

Stocked According To Plan

Dashboard use level

This indicator is recommended in dashboards used by national and sub-national managers.

Stocked According To Plan

Preconditions

This indicator is relevant in supply chains where:

There are established minimum and maximum levels for products for each health facility and store. Minimum stock level is considered the safety stock that is different from the reorder stock level. The maximum stock level is the safety stock plus the expected consumption between deliveries

Stocked According To Plan

System design

Relevant for supply chain systems with minimum stock level equal to safety stock. The indictor is not relevant in systems where the minimum stock level is considered equal to the reorder level, as the stock is expected to go below the minimum stock level. In these systems, alternative indicators such as Full Stock Availability and Closed Vial Wastage may be better employed.

Stocked According To Plan

Data needed

Stock balance
Minimum and maximum levels

Stocked According To Plan

Data sources

Physical inventory count
Stock cards/ledgers
Logistics management information system (LMIS

Stocked According To Plan

Data collection method

Stock balances should be collected at least twice per resupply period: just after and before delivery, to provide the highest and lowest stock balances in the resupply period.

Stocked According To Plan

Calculation

Stocked according to plan is determined by comparing the stock balance (stock on hand) to the established minimum and maximum levels to identify which products have stock balances below, within or above the recommended levels. Stocked according to plan occurs when the stock balance is between the set minimum and maximum stock levels, which are typically set by national policy, for instance regarding the number of months of stock to be held in each type of store or facility.

In a store or health facility, each product can be assessed as stocked according to plan. Alternatively a set of tracer products can be considered. When aggregating the indicator at higher levels, then a health facility or store is considered stocked according to plan if all vaccines and immunization supplies are stocked according to plan.

% of products stocked according to plan (in health facility or store) = (# vaccines stocked according to plan for all or a set of tracer products)/(total # products) x 100

% of health facilities stocked according to plan = (# health facilities stocked according to plan for all or a set of tracer products)/(total # health facilities) x 100

Examples
In health facility A, the inventory policy for all vaccines is:

Minimum level: 50 doses
Maximum level: 100 doses

The table shows the facility’s actual stock balances at the beginning and near the end of the supply period.

% of products stocked according to plan = 3/6 x 100 = 50%

The health facility would be considered not stocked according to plan, since not all products were stocked within minimum and maximum stock levels.

District A has seven health facilities that monitor stocked according to plan (SATP) at the end of each month. The district has quarterly resupplies, so the indicator is reported quarterly. Six tracer products are used to monitor stocked according to plan.

% health facilities SATP (July) = 5/7 = 71%
% health facilities SATP (Q3) = 2/7 x 100 = 29%

The monthly calculations show that by the end of the resupply period, health facilities that were stocked according to plan in the beginning of the resupply period (right after supplies were received) reach below the minimum stock levels for one or more of the tracer products by the end of the resupply period. Overall for the quarterly resupply period, only 29% of health facilities were stocked according to plan.

Stocked According To Plan

Visualization and interpretation

The stock balances in stores can be visualized in charts, such as the two below. For easier interpretation of the SATP visualization, the minimum and maximum stock levels are included in the graphs. In health facilities, a simple graph can be used to track the stock levels, whereas in the national store, a sophisticated graph that predicts the future stock levels based on consumption can be used. Either graph is equally useful if drawn manually on a paper chart.

Visibility for Vaccines (ViVa) prototype, UNICEF.

The above chart indicates that stock levels exceed the maximum level after each delivery, but then return to stocked according to plan during the resupply cycle. There is therefore minimum risk of expiries, but delivered quantities could be reduced and delivered to other facilities if resources were limited.

When considering the aggregated reporting, a stacked bar chart or a table could be used. The stacked bar chart shows the number of health facilities in a district that are stocked according to plan (green) and above (red) or below (blue) the recommended stock levels.

The table shows the percentages of health facilities stocked according to plan and the stock balances of the recently introduced IPV vaccine. A table is also recommended for use at the national level. To keep the table simple to read, stock balances for only a limited number of vaccines (e.g., vaccines currently being introduced) are included.

Stocked According To Plan

Potential corrective actions

Prioritize actions for critical or problematic products and/or locations with low stocked according to plan percentages.
Verify stock level excursions outside of the stocked according to plan interval.
Perform root cause analysis to identify the reasons for under- or oversupply. Analysis should account for the time of measurement relative to stock receipt (i.e., stock levels should be at or slightly exceed the maximum upon stock receipt and decrease over time).
Review and revise inventory and distribution policies including minimum and maximum levels

Stocked According To Plan

Related indicators

Closed Vial Wastage
Functional Status of Cold Chain Equipment
Full Availability
On-Time and In-Full Delivery
Cold Chain Capacity Utilization

Temperature Alarm Rate

Used as a proxy for measuring vaccine potency and safety. Exposure to temperatures outside this range indicates a risk of heat or freeze damage to sensitive vaccines.

The following questions can be answered by monitoring this indicator:

Is cold chain equipment functioning properly?
Is there risk of heat exposure to vaccines?
Is there risk of freeze damage to vaccines?
Which CCE devices are in need of repair or replacement

Expand this sheet Close this sheet

Temperature Alarm Rate

Definition

Number of times the temperature inside cold chain equipment (CCE) exceeds or drops below a reference range. The indicator is applicable where vaccines are stored and during transportation. CCE is defined as all refrigerators, freezers, passive storage devices, and walk-in cold rooms and freezer rooms designated for storing vaccines.

Temperature Alarm Rate

Performance objective

Potency

Temperature Alarm Rate

Domain

Cold chain management

Temperature Alarm Rate

Full indicator name(s)

Rate of heat and cold alarms per monitoring period (e.g., per month)
Number of CCE devices with more than a certain number of temperature alarms during a monitoring period

Temperature Alarm Rate

Dashboard use level

This indicator is recommended in dashboards used by sub-national and national managers. Visual monitoring of temperature (i.e., through monitoring of 30-day temperature recorders and/or thermometers) is recommended in health facilities and stores.

Temperature Alarm Rate

Preconditions

The indicator is relevant for all types of immunization supply systems, for all locations where immunization products are stored. A mechanism for routinely measuring and recording temperature is needed in each device designated for storing vaccines.

Temperature Alarm Rate

System design

Relevant in all types of logistics systems.

Temperature Alarm Rate

Data needed

Continuous or point-in-time temperature readings recorded over a time period. Continuous temperature monitoring is highly preferred, since it allows greater accuracy in detecting temperature fluctuations. For primary and sub-national stores, programmable electronic temperature and event logger systems are the best option. In smaller stores and health facilities, 30-day electronic temperature records with a stem thermometer as a backup are considered best practice. A stem thermometer alone only indicates the temperature at the time a reading is taken, which is no more than 14 times per week. A 30-day temperature logger takes at least a thousand readings per week.

The number of excursions, or alarms outside the designated temperature ranges, is needed. Alarm thresholds are set by WHO:

•   An excursion is defined as any event during which the temperature inside the cold chain equipment goes below 2° C or above 8° C.
•   A high temperature alarm is defined as any event during which the temperature goes above 8° C for 10 continuous hours.
•   A low temperature alarm is defined as any event during which the temperature goes below −0.5° C for one hour.

For locations measuring and recording temperatures manually twice daily, alarms may be difficult or impossible to record. Record any excursion outside the range of 2°C to 8°C for refrigerators and −15°C to −25°C for freezers. A point-in-time ‘temperature in range’ indicator may be used instead. However, a point-in-time temperature reading within temperature range does not provide any indication about temperature excursions that may have occurred at other times throughout the day when the temperature was not being recorded (e.g., a cold exposure overnight, when ambient temperatures dropped). Note that WHO no longer recommends stem thermometers and point-in-time recording of temperature as the primary means to monitor temperature in cold chain equipment.^3

2World Health Organization, How to Monitor Temperatures in the Vaccine Supply Chain: WHO vaccine management handbook – Module VMH-E2-01.1, WHO, Geneva, 2015, <http://apps.who.int/iris/bitstream/10665/183583/1/WHO_IVB_15.04_eng.pdf?ua=1>, accessed 7 November 2015.³World Health Organization, ‘The Vaccine Cold Chain’, Module 2 in Immunization in Practice, WHO, Geneva, p. 22, <www.who.int/entity/immunization/documents/iip2014mod2aug4.docx?ua=1>, accessed 7 November 2015.

Temperature Alarm Rate

Data sources

High/low temperature alarms (built into CCE or temperature monitoring devices)
Continuous temperature recording devices, including 30-day temperature recorders. Wherever possible, temperatures should be recorded automatically.
Proof of delivery (POD) for measuring temperature during transit if a temperature recording device is included

Temperature Alarm Rate

Calculation

Temperature alarm rate = number of high and low temperature alarms per reporting period

This indicator can also be calculated using the number of CCE devices with more than a set threshold of temperature alarms in a given period.

It can be further broken down by reasons for alarms (if known) or into ’resolved‘ and ’unresolved‘ alarms. That is, an alarm due to a resolved power outage would not be treated the same as an alarm due to mechanical problems. The alarm rate can also be disaggregated by facility, by device or by device type (make, model, energy source, etc.) to monitor performance.

Examples
A facility has one ice-lined refrigerator with a 30-day temperature logger. During a
supervisory visit, the temperature data is downloaded, and the following temperature alarms are noted:

This facility had an alarm rate of 2 alarms per month during the month of April. Alternatively, heat and cold alarms can be reported separately, with an alarm rate of 1 alarm per month for each. If the cause of alarms is known, this indicator can be further disaggregated (e.g., 1 heat alarm due to power outage).

In a district comprising 40 health facilities, each of which is using 30-day temperature recorders, there were a total of 16 alarms during the past month; 4 high temperature alarms and 12 low temperature alarms.

The rate is reported as 4 high temperature alarms per month and 12 low temperature alarms per month or 16 temperature alarms per month.

Temperature Alarm Rate

Visualization and interpretation

Continuous temperature recording devices can provide tabular readouts of temperature data, including alarms.

Across facilities, the alarm rate can be displayed in a colour-coded table to highlight facilities with frequent temperature excursions. These may be targeted for repair or replacement of CCE or for additional training on vaccine management.

Listing or visualizing only poorly performing fridges or facilities can allow for easier prioritization of facilities that need immediate attention.

The overview below shows the alarms for six CCE devices over a month (January). The graph highlights the high and low temperature alarms and the percentage of time that the CCE was within the recommended temperature range during the month. It provides a quick overview of CCE performance. This is an advanced type of visualization that requires continuous temperature monitoring.

Nexleaf dashboard snapshot, based on Project Optimize excel tools.

There are simpler ways to visualize CCE with and without alarms. Across multiple health facilities in a district, the percentage of CCE with or without alarms can be displayed as a pie chart and trends can be tracked using a line graph.

Further disaggregation by location where the alarms are taking place, as well as considerations related to the value of vaccines at risk should be taken into account. For instance, more urgent action might be needed for a walk-in cold room storing thousands of doses of vaccines than for a single refrigerator at health facility level.

Temperature Alarm Rate

Potential corrective actions

Ensure that facilities follow standard operating procedures through supportive supervision. For instance, facility staff should remove vaccines from CCE not maintaining temperature within recommended ranges in accordance with contingency plans and should discard vaccines that have VVM stage 3 or 4 and vaccines that fail the shake test.
Determine cause of equipment dysfunction; solicit repair or replacement of non-functional equipment
Ensure that contingency plans are in place for all facilities
Perform regular routine maintenance of all CCE to prevent future breakdown
Train facility staff to improve inventory management practices
Use temperature alarm profiles of various types and models of CCE to inform procurement
Use temperature alarm profiles to plan for repair and replacement of CCE

Temperature Alarm Rate

Related indicators

Cold Chain Capacity Utilization
Number of Maintenance Visits, Requests and Repairs
Cold Chain Equipment Functioning
Cold Chain Equipment Uptime
Mean Time to Repair Cold Chain Equipment
CC Energy Source Report
Number or % of Vaccines Discarded Due to Heat Exposure or Freeze Damage