Sign In
TechNet-21 - Forum
This forum provides a place for members to ask questions, share experiences, coordinate activities, and discuss recent developments in immunization.
Discussions tagged Temperature monitoring
Upcoming webinar on VVM-based vaccine management - January 21st @ 4pm CET
Please join us for a webinar on vaccine vial monitor (VVM)-based vaccine management, on the occasion of publishing of “The Book of VVM: Yesterday-today-and-tomorrow” by Umit Kartoglu. The webinar will take place on Tuesday, January 21st at 4pm CET. During the webinar, the presenters will discuss the critical role of VVM in vaccine management policies as well as the VVM shaping the future of the (digitized) cold chain. The presenters will be:
Umit Kartoglu, Co-founder and CEO, Extensio et Progressio, Collonge-Bellerive, Switzerland (author of The Book of VVM)
Debbie Kristensen, Director of Vaccine Technology, Strategy, and Policy / Medical Devices and Health Technologies, PATH, Geneva, Switzerland
Ted Prusik, Co-founder and Senior Vice President, Temptime Corporation, Morris Plains, NJ, USA
For presenter biographies, refer to the attached document.
It is not an overstatement to say that VVM is one of the most important recent innovations in the area of public health, providing health workers with advisable and dependable means of ensuring that the administered vaccine has not been damaged by heat. Amongst all the temperature monitoring devices and tools, VVM is the only one that has dramatically changed the course of vaccine management practices as well as shaped the future of the cold chain. Some critical approaches we have today in vaccine management have only been made possible with the help of VVM, and others have been made more effective.
For a quick review of VVM-induced vaccine management policies and VVM based vaccine management, refer to pages 201 and 245 in The Book of VVM (free download in PDF and ePUB3 formats from http://kartoglu.ch/vvm).
Registration at: https://who-meeting.webex.com/who-meeting/onstage/g.php?MTID=e27289500ca959052459c45ec0e2540e1
REPOST: Remote Temperature Monitoring Devices (RTMDs) User Survey
Denise
In
WHO PQS is conducting a survey to learn from users of Remote Temperature Monitoring Devices (RTMDs) and inform the future of vaccine cold chain monitoring.
Have you ever…
Received a text, email or notifications from an RTMD?
Interacted with RTMD hardware in the field, including responding to alarms?
Setup RTMD hardware and/or software?
Interacted with an online RTMD web portal?
Conducted repairs or service on RTMDs?
If so, we want YOU to help inform the future of RTMDs!
Interested? Please send an email to dhabimana@path.org to receive the survey link!
Best wishes,
Denise Habimana
Vaccine Cold Chain Technologies Program Officer at PATH
WHO PQS Manufacturer Consultation Report
Denise
In
Dear TechNet Community,
On Tuesday 10th July 2018, the WHO Performance, Quality and Safety (PQS) Working Group conducted its first face-to-face technical consultation with manufacturers of PQS prequalified immunization products and devices from the E003 (Refrigerators and freezers) and E006 (Temperature Monitoring Devices) equipment categories - some of you may also have followed the event in real time via the TechNet-21 Twitter account.
Thirty industry representatives took part in the meeting, alongside ten non-industry members and partners of the WHO PQS Working Group, to discuss and agree on a path forward for some significant proposed changes to immunization cold chain equipment standards.
The PQS Manufacturer Consultation proved to be a highly successful first step towards greater collaboration with manufacturers on PQS standards-review and standards-setting processes. In particular it provided manufacturers with greater visibility on in-country realities and complexities and helped solicit industry insights and inputs on technical and product solutions to meet varying challenges in EPI program environments.
More insights on the themes covered during the meeting as well as key discussion points and meeting outcomes have been provided in the PQS Manufacturer Consultation Report. The report can be accessed in the TechNet Library by clicking on the following link: https://www.technet-21.org/en/library/main/4926.
Enjoy the read!
On behalf of the PQS Team,
Denise
Remote Temperature Monitoring Devices (RTMDs) User Survey
Denise
In
WHO PQS is conducting a survey to learn from users of Remote Temperature Monitoring Devices (RTMDs) and inform the future of vaccine cold chain monitoring.
Have you ever…
Received a text, email or notifications from an RTMD?
Interacted with RTMD hardware in the field, including responding to alarms?
Setup RTMD hardware and/or software?
Interacted with an online RTMD web portal?
Conducted repairs or service on RTMDs?
If so, we want YOU to help inform the future of RTMDs!
Interested? Please send an email to dhabimana@path.org to receive the survey link!
Best wishes,
Denise Habimana
Vaccine Cold Chain Technologies Program Officer at PATH
Finding the Perfect Temperature for Vaccines
Finding the Perfect Temperature: Protecting Lifesaving Vaccines in Remote Areas
Immunization is one of the most successful and cost-effective public health interventions in history, saving 2-3 million lives every year. In Tanzania, JSI is installing remote temperature monitoring devices to strengthen the immunization cold chain and ensure that vaccines make it to the last mile. Click here to view the photo essay.
Immunization is one of the most successful and cost-effective public health interventions in history, saving 2-3 million lives every year. In Tanzania, JSI is installing remote temperature monitoring devices to strengthen the immunization cold chain and ensure that vaccines make it to the last mile. Click here to view the photo essay.
DIGITALLY MIMICKING THE ANALOGUE FUNCTION OF A VVM
Imagine that VVM’s do not exist. As an alternative a tiny electronic device could be developed which would record the temperature history of the vaccine vial. The device would be attached to the vial and graphically display the temperature history of the vaccine. If a health worker wanted to know how the temperature history of the vaccine would affect its life, how would they do it? This is essentially the same problem faced when trying to interpret the data obtained from a refrigerator or vaccine carrier data logger. Currently there is no analytic method to determine how temperature excursions affect the life of a vaccine.
VVM’s
A VVM may be thought of as an analogue device whose function is governed by the Arrhenius rate equation. The output of this elegantly simple device is a displayed dot of varying density. The density of the dot indicates the percent of life left in the vaccine. If VVM’s were not available the electronic device attached to the vaccine vial could use the collected data to calculate how temperature variations affected the life of a vaccine. This calculation could be done with the aid of the Arrhenius rate equation. The display of this device could mimic a VVM by displaying a dot of varying density to indicate how much vaccine life is left. However with digital electronics it may be more informative to display the number of days of life left. This same analytic tool could be used to interpret the data recorded on a refrigerator or vaccine carrier data logger. This would be a major step forward since there is no analytic method currently in use to determine how temperature excursions effect the life of a vaccine.
ALARM SYSTEMS
Knowing how temperature excursions affect the life of a vaccine can lead to more intelligent and responsive alarm systems. WHO recommends that an alarm is turned on when temperatures rise above 8C for 10 hrs. This recommendation could be misleading an alarm could be turned on when there are no problems and not turned on when there are serious problems. For example, for a 24 hours day an excursion to 9C for 11 hrs., then a return to 5C for 13 hrs. would turn an alarm on however this excursion would not lower the expected life of a vaccine. On the other hand if each day the temperature climbs to 20C for 9 hrs. in the afternoon, and than returns to 5C for the remainder of the day the alarm would not be activated, however the life of the vaccine would be cut in half.
A DIGITAL ANALYSIS OF TEMPERATURE DATA BASED ON THE ARRHENIUS RATE EQUATION WOULD HAVE THE FOLLOWING ADVANTAGES:
Simplified data interpretation: The analysis would show the effect of a temperature excursion in a simple to interpret form.
The severity of a refrigerator problem would be indicated which is a valuable piece of information for management.
Elimination of false alarms: Alarms would coincide with indication from a VVM.
More responsive alarm systems.
Reduced vaccine waste.
Data to show that outreach programs can be safely extended.
Simpler and less expensive designs for refrigerators without sacrificing vaccine life
These advantages could come to fruition if we simulated digitally what a VVM does in an analogue fashion.
Realtime Cooler Temperature & Location Monitoring in the Field
All,
I am trying to solve the following problem...
We have small medical teams that travel to remote locations, to deliver vaccines to patients.
Many of the routes are through jungle, mountains, dessert, etc and take several hours or longer to reach to destination.
The teams use motorbikes, and walking through rough terrain to get to the destination.
During the traveling, there is no way to tell what's going on in the cooler until the team reaches the destination.
We've been asked to:
Track cooler location in real time
Track cooler temperature in real time
This looks like a solvable IoT problem.
Look forward to getting experiences and insights on this.
Thanks
Integrating Cold Chain Data into OpenLMIS: A Collaboration with Nexleaf Analytics
OpenLMIS
In
Greetings,
A new blog is available on openlmis.org, presenting an exciting collaboration between OpenLMIS and Nexleaf Analytics to incorporate remote temperature monitoring (RTM) data into the OpenLMIS logistics software: http://openlmis.org/integrating-cold-chain-data-openlmis-collaboration-nexleaf-analytics/
This collaboration allows anyone adopting OpenLMIS and utilizing any RTM system to have an informed view of the cold chain status alongside their core vaccine stock and delivery data.
Integrating data systems can reduce data entry, improve data quality and usability, and accelerate insight generation. Combining datasets from different sources provides new opportunities for analysis and insight into the supply chain and streamlines processes for health workers and decision-makers alike. The interfaces developed by OpenLMIS and Nexleaf are open and standards based. This means that any provider of RTM hardware or data analysis systems will be able to integrate with OpenLMIS using the interfaces being developed by this project. This work is designed to benefit all manufacturers and providers of RTM hardware and software systems.
We look forward to your feedback on the article. Happy reading and best wishes in your work.
Tenly Snow
OpenLMIS Community Manager
OpenLMIS Community Manager
Temperature monitoring devices and Thermostat for vaccine refrigeration system
Dear Technet members,
I would like to share my thoughts on the temperature monitoring devices and thermostal for vaccine refrigeration system.
In the early days of EPI, temperature of the refrigerator was measured and displayed by a simple analogue thermometer which was either a bi-metal or stem type alcohol thermometer. Twice daily the health worker records these temperature on the temperature monitoring sheet and posted on the door of the fridge. Filled temperature records are collected, kept on the file at the end of each month and replaced by a new one. Temperature recorded was the temperature obtained during that time when it was read. There was no information on the temperatures during the night or between the time interval when temperatures were obtained. These old thermometers need to be calibrated more often which was a big challenge at the remote health facility.
To rectify the problem an electronic thermometer that could continuously record and display the temperature of the fridge was developed. It does not need calibration till its lifetime of 2-3 years when it has to be replaced. Average temperature readings could be read for the past 30 days. The device was further developed to include a usb connector and with temperature alarm alert and temperature data could be downloaded for record and review purposes.
The latest temperature monitoring and recording device has a temperature sensor placed inside the fridge and could send the temperature readings on a computer/cellphone and could further be accessed via cloud etc.
All of the above temperature monitoring and recording devices are passive devices, in the sense that it does not take any corrective action if temperature excursions are encountered. It is simply used to inform or alert the health worker that something is wrong with the fridge. Corrective actions at health facility level relies on the health worker who is alone and frequently busy and not specifically trained to do the job of diagnosing the cause of the problem. With shortage of the technical staff at the national/provincial/distric level, it is not expected to rectify the problem as early as it is needed thus causing potency problem to the vaccines, missed vaccine sessions and increase unopened vial vaccine wastage.
Allow me to give information to each one who are not familiar with refrigeration system, that every vaccine refrigerator/freezer (with the exception of the absorption kerosene refrigerator) has a temperature control which is commonly called a “Thermostat”. Thermostat is an active device (compared to a thermometer which is a passive device) which is connected in the electrical circuit component of the refrigerator. Its main job is to regulate or control the temperature of the refrigerator according the temperature setting by the user. The old thermostat control switch is numbered starting from 0 to say 5. Each number has a corresponding cut-in and cut- out temperature.
How does thermostat work?
The purpose of all refrigeration system is to lower the temperature of the confined space below that of its immediate surrounding. The confined space of the refrigerator is the vaccine storage compartment. The refrigerant compressor of the refrigeration system once connected to the grid or solar panel will continuously run to lower the temperature of the vaccine storage compartment thus putting the freeze-sensitive vaccines at risk. Running compressor motor can be stopped by any of the following actions: a) if power from the grid is manually disconnected by the user or accidental power shutdown; b) the overload protector of the compressor is automatically activated due to extreme heat and voltage fluctuation; and c) the thermostat which was set at a desired temperature is automatically activated. Please note that once the compressor is stopped by either of the above ways the temperature inside the vaccine storage compartment will continue to rise. Operation of the refrigerator will commence again if either the refrigerator is manually connected to the grid, the overload protector cools down and voltage is back to the normal range or the thermostat automatically reach its cut-in set point. Therefore, THERMOSTAT is an active device that automatically regulates the temperature between the cut-in temperature of +8°C and the cut-out temperature of +2°C of the vaccine storage compartment. Cut-in temperature of +8°C is the temperature when refrigerator compressor motor will start and cut-out temperature of +2°C is the temperature when refrigerator compressor motor will stop to protect freeze sensitive vaccines from freezing.
It is therefore very important call to action for the refrigeration system equipment manufacturer to further conduct a very thorough study to equip the vaccine refrigerator with a very accurate and precise thermostat control device that will work at various conditions in the field.
WHO PQS has introduced new standards for refrigerators to have a sealed thermostat. This effort is highly appreciated hoping to reduce unopened vaccine vial wastage due to freezing temperature in the refrigerator. If thermostat is not sealed untrained staff as in the previous refrigeration units will try to adjust the temperature setting of the thermostat thus placing the both freeze sensitive and not freeze sensitive vaccines at risk. Electronic sealed thermostats are now used by equipment manufacturer. Temperature settings of these thermostat can only be altered by a manufacturer trained technician.
Recently, countries are receiving refrigeration equipment under the GAVI CCEOP project. Equipment are laboratory tested and PQS compliant, however their performance could actually be tested at different and difficult environment and situations in the field. To be able to assist the equipment manufacturer to develop a more accurate and precise thermostat, country’s EPI national program should monitor equipment performance and share feedback to TechNet for follow up actions.
Hope this helps.
Alejo H Bejemino
Consultant
Medical Physics and Health Physics Services, Inc
Philippines
Immunization Academy Update
This month several new data monitoring and supply videos have been added to www.immunizationacademy.com, including nine new training videos in Swahili. Anyone working in support of EPI can be notified when new videos are added via WhatsApp. Using WhatsApp, text "join" to +255 765 578 712 and you will be added to the Immunization Academy broadcast list. The Immunization Academy is a BMGF-sponsored initiative that provides health professionals with instant access to a comprehensive library of short, practical videos to support immunization training and delivery.
Introducing data-driven, remote oversight (Vaccine Supply Chain Futures 4/6)
John Lloyd
In
This is the fourth topic in my six-part Vaccine Supply Chain Futures series (Please see the attached file for the full text) focused on the introduction of a system of remote temperature monitoring integrated with equipment maintenance management.
Remote temperature monitoring systems represent an opportunity for a system-wide improvement in the performance of equipment maintenance and consequently, more reliable vaccine distribution. Remote temperature monitoring automatically transmits data to a central server where it is analysed and disseminated to managers. This is a significant step forward compared to stand-alone temperature recording in each refrigerator. It replaces manual recording and paper reporting systems that have always suffered from lack of compliance and inaccuracy due to incomplete, delayed and faulty data aggregation.
Remote Temperature Monitoring is a means to:
Supervise health facilities to maintain correct storage temperatures,
Track the efficiency of equipment maintenance at local level,
Country wide data on the performance in use of each model will inform the choice of equipment for procurement to maximize reliability,
Suitably anonymous versions of the country data will be used at international level to inform equipment manufacturers and regulatory experts on the rates of failure and their principal causes.
Please reply to this post with your views and I would be grateful if you could answer just four questions in this SurveyMonkey:
https://www.surveymonkey.com/r/L38J79Q
I will return the analysis to you at the end of this week and the results will be posted before the TechNet Conference for all five topics.
Please see the attached file for the full text.
Thank you!
Transportation of vaccines from the cold-chain point of a planning unit to the outreach session sites – A model.
Dear viewers - Please find the following and the attached illustration for a critical review and needful.
Background:
CCH Mrs Anitha Beth and Asst CCH Mr Mukesh could not believe when the author shared his observations that the “T” series vaccines were found in “Frozen” condition at the session site far off from the ILR point. Temperature record book never showed subzero recording on any day. CCH had to distribute vaccines to ~50 session sites on Thursdays and ~30 session sites on Saturdays. In a hurry, unconditioned icepacks were loaded in one of the vaccine carriers at about 7 AM and reached outreach at about 10 AM. Jerky movement during the excursion hastens the process of freezing. All vaccination service providers are aware that frozen “T” series vaccines lose potency and the precipitated particles act as foreign body and may result in cold sterile abscess.
As per the formal training, vaccine vials are to be kept in a Ziploc pouch and placed in the vaccine carrier packed with 4 conditioned ice-packs. In northern states of India, with more than 2Lakh population per planning unit and ~300 outreach sessions per month, often / accidentally unconditioned ice-packs may be placed as in the above instance.
Other common / frequent observations were unclean vaccine carrier, ill fitting lid, no strap to keep the lid in-situ, tampered vaccine carrier, vaccine vials without label, VVM in discard stage, wrong diluent or no diluent, only BCG/Measles/MR/JE, no dropper, only OPV, diluent outside the cold chain in the vanity bag, no starting date on vaccine vials of Open Vial Policy [MDVP] and the like.
And now: Recently, on 21st April 2017, a senior consultant from New Delhi also being directed by his superiors from an international RI development partner agency visited the dedicated vaccination clinic of our KVG Medical College to address above issues, especially to avoid freezing of vaccines in the vaccine carrier while transporting.
In response to this I wish to share the following, practicing in all the 3 Planning Units attached to our college with illustrations.
Transparent plastic container has a screw cap, tapering towards the bottom and is icepack height. Holes are made using heated tip of piston of 2mL glass syringe for sustaining uniform temperature both inside and outside the plastic container within the vaccine carrier.
However; inspite of training, orientation, re-orientation; passionate Sustained supportive supervision is indispensable, helps in hands on training, maintains perfection of the programme.
Anticipated benefit: Apart from ensuring delivery of quality vaccine, this incurs negligible expenditure, aesthetic, easy to operate and can save millions of $ on operational research of designing new generation vaccine carrier.
Sharing with the viewers for comments / sharing their experiences in this field.
Cold room temperature mapping
Dear All:
how can we conduct study of cold room temperature mapping and how to analyse the data collected?
Cartographie de la température de Chambres froides au Niger
Bonjour
Le Niger a entrepris en 2016 avec l'aide de ses partenaires la cartographie de la température dans ses Chambres froides. Nous avons beaucoup appris de cette expérience notamment sur :
- La cartographie de toutes les chambres froides qu'elles soient anciennes ou nouvelles
- Le réglage de la consigne d'arrêt des unités de réfrigeration
- La fiabilité du Kit Berlinger (SmartView)
Nous proposons les recommandations suivantes à d'autres pays intéressés par la cartographie de température :
- Former les formateurs qui doivent poursuivre la cartographie dans le reste du pays
- Doter l'équipe pays d'un kit de documentation et de matériels (kit log tag)
- Installer des dispositifs de monitorage de température à distance dans toutes les chambres froides aux niveau central et régional.
J'aimerai partager avec vous le resumé de cette activité dans le doucument attaché. Pour plus d'information ou de questions, veuillez contacter Abdoulaye Boubacar, Chef Service Logistique à la Direction des Immunisations, Niger, abdouboubacar01@yahoo.fr ou Cheick Tidiani Coulibaly Conseiller Technique, JSI Research & Training Institute, Inc. cheick_coulibaly@ne.jsi.com
“CROWD TESTING” An Easily Implemented Method of Ice Pack Conditioning
Vaccines can tolerate temperatures above 8 deg C however temperatures below 0 deg C will freeze a vaccine and it will lose it’s efficacy. At temperatures above 8 deg C the life of a vaccine will be lowered but the exposure will not be catastrophic. For the most sensitive vaccine, VVM II, the life of the vaccine will decrease from 104 days to 99 days when exposed to 20 deg C conditions for one day. For a VVM7 vaccine exposed to the same one day temperature excursion the life of the vaccine will decrease from 650 days to 643 days. As Umit points out in a recent post [High Temperature Freezers], it is sometimes logistically difficult to get ice packs conditioned for an early morning departure. SDD freezers do not cool at night and as a consequence in the morning the ice packs are conditioned or close to being conditioned. Similarly, if an ice pack carrier is loaded with ice packs the night before a trip the ice packs will be conditioned or near conditioned the next morning. If the ice packs are not completely conditioned, they could be placed in the carrier earlier in the day to increase conditioning time.
[b]Crowd Testing[/b]
If a reader is involved in vaccine distribution this technique can easily be tested. In the afternoon, cold ice packs would be placed in a carrier along with a thermometer. The next morning the temperature of the carrier would be recorded and the ice packs examined. Water in the ice packs indicate that the ice packs are conditioned. It is most critical that the ice packs closest to the vaccines are fully conditioned. If the ice packs are not conditioned, repeat the experiment with the ice packs placed in the carrier earlier in the day. If, for example, the ice packs are conditioned 5 extra hours over the minimum conditioning time to reach 0 deg C, the consequence would be minimal. The length of time acceptable temperatures are maintained would be only reduced by 5 hours. Using “Crowed Testing” the effectiveness of this technique could be easily determined.
If you are involved in the distribution of vaccines, I would appreciate your help in evaluating this method. Please send your answers to the following questions to Tech Net
Name
Location
Date
Model of carrier
Number and size of packs in carrier
Ice pack freezer temperature
Number of hours in carrier over night
Average ambient night time temperature
Temperature of the carrier in the morning
Is there water in the ice packs?
Do you estimate more or less time is needed for conditioning?
Comments
Passive storage devices--new technology requires new SOPs
Does anyone have experience with vaccine passive storage devices in the field? We are piloting the PQS pre-qualified Arktek device in three rural health centers. By design, the device maintains a temperature range between 0°C and 10°C which will not freeze the vaccines so should not pose any risk to the potency of the vaccines. We are seeing some unease from health workers and EPI managers, though, who are concerned that vaccines would be out of the 2°C – 8°C range and at risk of freezing, even though all the evidence shows this type of passive storage device will not freeze vaccines when following proper procedures for conditioning the ice.
On the one hand, this reaction from the health workers speaks to the success of training and insistence on good temperature control between 2 and 8°C. On the other hand, this new technology requires changes to standard operating procedures that have not been fully vetted with stakeholders and still require education and updating.
Has anyone else had experience with this? Any suggestions on how to approach these types of changes to procedures required for this new technology?
I appreciate any thoughts.
Wendy
Refrigerated truck monitoring systems - Measuring interval of logging
Dear Colleagues,
Could you please share any links for documents (regulations or guidelines) where disscussed measuring interval for refrigerated truck monitoring systems (print-paper loggers).
Оr please share practice in your country.
Thank you in advance,
Regards
Could you please share any links for documents (regulations or guidelines) where disscussed measuring interval for refrigerated truck monitoring systems (print-paper loggers).
Оr please share practice in your country.
Thank you in advance,
Regards
Introduction of the new temperature monitoring website
The WHO UNICEF immunization Supply Chain Hub (iSC Hub)has released a new temperature monitoring website hosted on TechNet-21.
A stringent system for temperature surveillance is essential to maintain the quality of the vaccines in an effective Supply Chain. By preventing vaccine exposure to freezing or hot temperatures in the cold chain, a functioning temperature monitoring system helps ensuring vaccine potency, avoiding both wasting vaccine investments (notably through reducing closed vial wastage) and potentially placing children at risk of contracting vaccine preventable diseases.
This new areaof the TechNet-21 website stresses that effective surveillance depends on the development of a system of accountabilities, processes and equipment (in this order or importance) in order to ensure that temperature excursions are detected and that appropriate corrective measures are taken to avoid damage to the vaccines.
This website presents technical and programmatic information on available solutions to maintain vaccines in adequate temperature range at fixed storage and brings forward country experiences both on the use of 30DTR devices and RTM technologies. It aims at:
Providing decision makers and stakeholders with pragmatic step-by-step guidance and material to build a Temperature Monitoring system;
Encouraging experience sharing in order to leverage good practices and useful tools among countries and partners wanting to introduce or to update systematic temperature monitoring based on some of the technologies described.
You can access the website through the following link: http://www.technet-21.org/en/resources/temperature-monitoring
We hope you will find it useful and look forward to any feedback you may have.
Adama E. Sawadogo
Lead of the iSC Hub Strategic Focus Area
On temperature monitoring
A stringent system for temperature surveillance is essential to maintain the quality of the vaccines in an effective Supply Chain. By preventing vaccine exposure to freezing or hot temperatures in the cold chain, a functioning temperature monitoring system helps ensuring vaccine potency, avoiding both wasting vaccine investments (notably through reducing closed vial wastage) and potentially placing children at risk of contracting vaccine preventable diseases.
This new areaof the TechNet-21 website stresses that effective surveillance depends on the development of a system of accountabilities, processes and equipment (in this order or importance) in order to ensure that temperature excursions are detected and that appropriate corrective measures are taken to avoid damage to the vaccines.
This website presents technical and programmatic information on available solutions to maintain vaccines in adequate temperature range at fixed storage and brings forward country experiences both on the use of 30DTR devices and RTM technologies. It aims at:
Providing decision makers and stakeholders with pragmatic step-by-step guidance and material to build a Temperature Monitoring system;
Encouraging experience sharing in order to leverage good practices and useful tools among countries and partners wanting to introduce or to update systematic temperature monitoring based on some of the technologies described.
You can access the website through the following link: http://www.technet-21.org/en/resources/temperature-monitoring
We hope you will find it useful and look forward to any feedback you may have.
Adama E. Sawadogo
Lead of the iSC Hub Strategic Focus Area
On temperature monitoring
Evaluating Vaccine Temperature Monitoring Systems via Fuzzy Analytic Hierarchy Process
Interesting study from Turkey aimed at providing an approach to determine the best temperature monitoring system for vaccine storage in a warehouse.
AbstractCold chain management is a specific part of supply chain management. Cold chain managementdeal with products that should be transported and stored in certain conditions. Monitoring thetemperature in cold chain is one of the most critical and important factors. Effective temperaturemonitoring systems are a must have for a successful cold chain management in companies, sodeciding on the best temperature monitoring system is a strategic decision. The aim of this study isto determine the best temperature monitoring system for vaccine storage in a warehouse. Whilethere are many different criteria for selecting among different temperature monitoring systems, amulti criteria decision making methodology is used and we evaluated four different temperaturemonitoring systems by using fuzzy analytic hierarchy process (FAHP). Fuzzy numbers are used forthe application of analytic hierarchy process (AHP), because fuzzy numbers enable the decisionmakers to deal better with the uncertain and vogue circumstances while selecting the besttemperature monitoring systems. Electronic thermometer data logger system, wired data loggertemperature monitoring system, wireless data logger temperature monitoring system and cloudbased data logger temperature monitoring system are chosen as alternatives. First, we determined 4 main and 14 sub-criteria that affect the selection process and then we applied fuzzy analytic hierarchy process for selecting the best temperature monitoring system. Cloud-based data logger temperature monitoring system is determined as the best alternative.
AbstractCold chain management is a specific part of supply chain management. Cold chain managementdeal with products that should be transported and stored in certain conditions. Monitoring thetemperature in cold chain is one of the most critical and important factors. Effective temperaturemonitoring systems are a must have for a successful cold chain management in companies, sodeciding on the best temperature monitoring system is a strategic decision. The aim of this study isto determine the best temperature monitoring system for vaccine storage in a warehouse. Whilethere are many different criteria for selecting among different temperature monitoring systems, amulti criteria decision making methodology is used and we evaluated four different temperaturemonitoring systems by using fuzzy analytic hierarchy process (FAHP). Fuzzy numbers are used forthe application of analytic hierarchy process (AHP), because fuzzy numbers enable the decisionmakers to deal better with the uncertain and vogue circumstances while selecting the besttemperature monitoring systems. Electronic thermometer data logger system, wired data loggertemperature monitoring system, wireless data logger temperature monitoring system and cloudbased data logger temperature monitoring system are chosen as alternatives. First, we determined 4 main and 14 sub-criteria that affect the selection process and then we applied fuzzy analytic hierarchy process for selecting the best temperature monitoring system. Cloud-based data logger temperature monitoring system is determined as the best alternative.
Temperature data logger training August 2016
Tom Lewis
In
Following the success of the first training session in December, 2015 on the use of the MULTiLOG2 Remonsys Ltd is planning to run a second training session in August this year.
Purpose and Objectives of the Course
The objective of this course is to equip delegates with the ability to install, configure and to operate the MULTiLOG2 system, to enable them to train others to understand the technology and how they can use it to make the best use of the recorded information to protect the vaccines in their care.
Methodology
The course will be taught in English
A mixture of demonstration and hands-on training
A participatory approach will be applied during training sessions.
Training notes, manuals and handouts will be provided
Delegates Profile
Delegates should be field officers, either from local, national, regional, or international non-governmental organisations, independent companies or free lancers working in the cold chain.
Time Frame and Venue
Date of training: Thursday, 25th and Friday 26th August, 2016
Duration of training: 2 days. Location for training: Oxford, United Kingdom
Note: Delegates may require a visa so please apply early to your local British Consulate
Training Costs
There is no cost for the training course but attendees are expected to cover their own travel and accommodation costs They should bring with them a laptop computer running Microsoft Windows 7 or newer.
For further information: -
Please register your interest as soon as possible with Remonsys Ltd
E: info@remonsys.com
T: +44 1993 886996
Purpose and Objectives of the Course
The objective of this course is to equip delegates with the ability to install, configure and to operate the MULTiLOG2 system, to enable them to train others to understand the technology and how they can use it to make the best use of the recorded information to protect the vaccines in their care.
Methodology
The course will be taught in English
A mixture of demonstration and hands-on training
A participatory approach will be applied during training sessions.
Training notes, manuals and handouts will be provided
Delegates Profile
Delegates should be field officers, either from local, national, regional, or international non-governmental organisations, independent companies or free lancers working in the cold chain.
Time Frame and Venue
Date of training: Thursday, 25th and Friday 26th August, 2016
Duration of training: 2 days. Location for training: Oxford, United Kingdom
Note: Delegates may require a visa so please apply early to your local British Consulate
Training Costs
There is no cost for the training course but attendees are expected to cover their own travel and accommodation costs They should bring with them a laptop computer running Microsoft Windows 7 or newer.
For further information: -
Please register your interest as soon as possible with Remonsys Ltd
E: info@remonsys.com
T: +44 1993 886996
Temperature sensor positioning in cold rooms
Joel Coddet
In
Hello everyone,
We will in a few days to install the MultiLog 2 central cold chain in Madagascar. We have all the equipment and installation guide. What is missing is a positioning guide of temperature sensors inside the cold rooms. If you have this information, thank you to share to me. On our side, we conducted a temperature mapping to help us in this position. Regards,
Joel CODDET
Cold chain and logistic
OMS/WHO Madagascar
We will in a few days to install the MultiLog 2 central cold chain in Madagascar. We have all the equipment and installation guide. What is missing is a positioning guide of temperature sensors inside the cold rooms. If you have this information, thank you to share to me. On our side, we conducted a temperature mapping to help us in this position. Regards,
Joel CODDET
Cold chain and logistic
OMS/WHO Madagascar
Remote temperature monitoring (RTM); Accelerate scale-up !
John Lloyd
In
It's a mystery to me, considering that manual temperature monitoring of vaccine stores using monthly charts and glass thermometers was the first MIS achievement of the EPI in the last 4 years of the 1970s - yet the first Remote Temperature Monitoring (RTM) studies were 10 years ago and only a handful of countries have implemented RTM today. The benefits of the manual monitoring in 1976 were simple, clear and motivating. Those of the RTM should be even more enticing. Temperature recording is automatic and does not suffer from low compliance and inaccuracy. Supervision dashboards can post the problems in realtime, enabling oversight to be timely and constructive. Repair technicians could be equipped with diagnostic tools to respond to SOS calls based on data on failure modes, ask Nexleaf and PATH. Supervisors will be able to track areas and stores that are strong or weak in vaccines and equipment care. And managers will be equipped to base the selection of new equipment models for procurement, on those that actually perform the best.
So, if the technology has arrived, where is the implementation? The CCE Optimization Platform expresses RTM as a 'required upgrade’ to the WHO, minimal monitoring norm; I see it as the way to achieve a step-change towards safer, more efficient vaccine supply chain (SC). In a word, improved oversight of the supply chain will enable vaccines to be used to the limits of thier stability.:
to interest the countries, they should be presented the ‘big-picture’ benefits of supervision oversight
to do this, program managers meetings could allow presentations and consultants could be trained
financing to scale up the effort should be sought for the whole region - otherwise, it will be too slow!
RTM planning templates could help quantify the costs and training burden fast, with just enough data
present/brief/train consultants now working in the region
Now, what do you say? Isn't RTM the first step towards 'end-to-end' supply chain monitoring? Importance = HIGH?
So, if the technology has arrived, where is the implementation? The CCE Optimization Platform expresses RTM as a 'required upgrade’ to the WHO, minimal monitoring norm; I see it as the way to achieve a step-change towards safer, more efficient vaccine supply chain (SC). In a word, improved oversight of the supply chain will enable vaccines to be used to the limits of thier stability.:
to interest the countries, they should be presented the ‘big-picture’ benefits of supervision oversight
to do this, program managers meetings could allow presentations and consultants could be trained
financing to scale up the effort should be sought for the whole region - otherwise, it will be too slow!
RTM planning templates could help quantify the costs and training burden fast, with just enough data
present/brief/train consultants now working in the region
Now, what do you say? Isn't RTM the first step towards 'end-to-end' supply chain monitoring? Importance = HIGH?
VVM colour change as a continuous progression
Dan Brigden
In
WHO has just released a new infographic that summarises how health workers can use vaccine vial monitors (VVMs) to decide whether or not to use a vaccine vial. Most notably, it presents VVM colour change as a continuous progression, rather than as four distinct stages.
A VVM is a chemical indicator label attached to the vaccine container (vial, ampoule or dropper) by the vaccine manufacturer. As the container moves through the supply chain, the VVM records its cumulative heat exposure through a gradual change in colour. If the colour of the inner square is the same colour or darker than the outer circle, the vaccine has been exposed to too much heat and should be discarded.
The main purpose of VVMs is to ensure that heat-damaged vaccines are not administered. The VVM status is also used to decide which vaccines can safely be kept after a cold chain break occurs thus minimizing unnecessary vaccine wastage. In addition, VVM status helps the user decide which vaccine should be used first – a batch of vaccine showing significant heat exposure should be distributed and used before a batch that shows lower heat exposure, even if its expiry date is longer.
Please use the new infographic in guidance and training materials, and share widely. The infographic is available in English and French on the WHO website:
http://www.who.int/immunization/programmes_systems/supply_chain/resources/tools/en/index4.html
A VVM is a chemical indicator label attached to the vaccine container (vial, ampoule or dropper) by the vaccine manufacturer. As the container moves through the supply chain, the VVM records its cumulative heat exposure through a gradual change in colour. If the colour of the inner square is the same colour or darker than the outer circle, the vaccine has been exposed to too much heat and should be discarded.
The main purpose of VVMs is to ensure that heat-damaged vaccines are not administered. The VVM status is also used to decide which vaccines can safely be kept after a cold chain break occurs thus minimizing unnecessary vaccine wastage. In addition, VVM status helps the user decide which vaccine should be used first – a batch of vaccine showing significant heat exposure should be distributed and used before a batch that shows lower heat exposure, even if its expiry date is longer.
Please use the new infographic in guidance and training materials, and share widely. The infographic is available in English and French on the WHO website:
http://www.who.int/immunization/programmes_systems/supply_chain/resources/tools/en/index4.html
Remote Temperature Monitoring Data for Cold Chain Maintenance
Dear TechNet Colleagues,
I would like to share with you a report on our latest cold chain maintenance investigation in Mozambique: "Why Fridges Fail Part 2: RTM Data for Maintenance".
We are beginning to document how remote temperature monitoring data and dashboards can be used by maintenance technicians to significantly improve cold chain performance.
This assessment follows up on the randomized control trial conducted in 2014-2015 with Village Reach, PATH, and the Ministry of Health in Mozambique which aimed to evaluate the impact of remote temperature monitoring (RTM) on vaccine cold chain equipment performance at the health facility level. That study showed that RTM with SMS alerts increased fridge uptime and reduced freezing (See Appendix I of the report for more details.)
However, even some fridges in the RTM intervention group had chronic problems and failed to achieve 95% uptime.
We launched this follow-up investigation to answer the question: Can maintenance + RTM lead to 95% uptime?
We were very happy to work with Soren Spanner, a global cold chain expert with decades of experience. Soren traveled to Mozambique from September 24 until October 14, 2015. He used ColdTrace data to identify 27 fridges that were failing and worked with the Ministry of Health maintenance technician to visit or remotely diagnose and fix as many fridges as possible. Soren remained active in fridge fixes, phone calls and follow-ups through the end of November.
Perhaps the most interesting finding was that distinct patterns in the RTM data (which we call “temperature data signatures”) can be used to remotely diagnose problems and enable remote fixes via phone calls to nurses in clinics. For the improperly adjusted fridges in this investigation, thermostat adjustments increased uptime by 30%. Today, the MOH technician continues to use the dashboard data to diagnose problems, call clinics to address the issues, and verify fixes by monitoring subsequent temperature data. We also learned that, without access to key spare parts/tools, even an expert technician cannot get all fridges to 95%: flat batteries on solar fridges were one such problem that Soren did not have the resources to fix. With RTM data, however, those responsible for cold chain maintenance can identify which fridges need help and determine where to focus resources.
This report features all causes of fridge failure that the Technicians encountered, several key screenshots from the ColdTrace web dashboard, and critical recommendations for how to procure for and improve cold chain maintenance capacity at every level of the health system moving forward. We hope you’ll find these results approachable and easy to understand. I would greatly appreciate your feedback on the report and I hope we can start a discussion here on how temperature monitoring data could improve maintenance. Thank you. Regards,
Shahrzad
However, even some fridges in the RTM intervention group had chronic problems and failed to achieve 95% uptime.
We launched this follow-up investigation to answer the question: Can maintenance + RTM lead to 95% uptime?
We were very happy to work with Soren Spanner, a global cold chain expert with decades of experience. Soren traveled to Mozambique from September 24 until October 14, 2015. He used ColdTrace data to identify 27 fridges that were failing and worked with the Ministry of Health maintenance technician to visit or remotely diagnose and fix as many fridges as possible. Soren remained active in fridge fixes, phone calls and follow-ups through the end of November.
Perhaps the most interesting finding was that distinct patterns in the RTM data (which we call “temperature data signatures”) can be used to remotely diagnose problems and enable remote fixes via phone calls to nurses in clinics. For the improperly adjusted fridges in this investigation, thermostat adjustments increased uptime by 30%. Today, the MOH technician continues to use the dashboard data to diagnose problems, call clinics to address the issues, and verify fixes by monitoring subsequent temperature data. We also learned that, without access to key spare parts/tools, even an expert technician cannot get all fridges to 95%: flat batteries on solar fridges were one such problem that Soren did not have the resources to fix. With RTM data, however, those responsible for cold chain maintenance can identify which fridges need help and determine where to focus resources.
This report features all causes of fridge failure that the Technicians encountered, several key screenshots from the ColdTrace web dashboard, and critical recommendations for how to procure for and improve cold chain maintenance capacity at every level of the health system moving forward. We hope you’ll find these results approachable and easy to understand. I would greatly appreciate your feedback on the report and I hope we can start a discussion here on how temperature monitoring data could improve maintenance. Thank you. Regards,
Shahrzad
Why Conduct Effective Vaccine Management (EVM) Assessment?
Abstract from an article written by our EPI Ghana team. Comments are welcome!
For the full article please go to: http://www.sciforschenonline.org/journals/vaccines/article-data/IJVI-1-104/IJVI-1-104.pdf
Background: WHO and UNICEF supported Ghana to conduct Effective Vaccine Management (EVM) assessments in September 2010 and October 2014 respectively with the view to strengthening vaccine and logistics management for immunization at all levels in the country. In both assessments, temperature monitoring, which is a critical indicator for vaccine potency and immunization programme effectiveness, got very low scores of 50% (2010) and 67% (2014) respectively at the national level and even lower at the regional level. The low scores of temperature monitoring at the national and regional levels in both assessments generated concern and interest for review of the results of the two assessments to determine some of the contributing factors considering Ghana’s advancement in immunization with support of the Gavi Alliance and also with twelve antigens in the routine immunization programme. Objective: The main objective of the review was to determine causes of the poor performance of the temperature monitoring and to devise strategies to improve on it. Methods: Desk review of relevant documents including 2010 and 2014 EVM assessment reports, the report of the 2012 comprehensive EPI Review and the status of implementation of recommendations of the 2010 EVM assessment and the 2012 EPI Review was conducted. Results: The documents review revealed that the 2010 EVM assessment and the 2012 EPI Review recommended installation of continuous electronic temperature monitoring devices in the national and regional vaccine stores but it has not been implemented. The same recommendation has again been made from the 2014 EVM assessment. Conclusions: There was no significant improvement in the scores of temperature monitoring between the 2010 (50%) and 2014 (67%) assessment because of the failure of the Immunization Programme to implement the recommendations of the previous assessment (2010) and the EPI Review (2012) which called for installation of the continuous electronic temperature monitoring devices in the national and regional vaccine stores. Not until the continuous electronic temperature monitoring devices are installed, the scores of temperature monitoring will forever remain low in any future assessment. The question is why use limited resources to conduct EVM assessment if the recommendations to address key challenges of the immunization programme are not to be implemented? Countries conducting EVM in future should make plans to implement the recommendations from the outcome. This is the only way there can be improvement in the immunization programme. Periodic EVM assessment is essential to keep the supply chain system at the required standard. Keywords: Effective Vaccine Management (EVM); EPI; Temperature monitoring; Ghana
For the full article please go to: http://www.sciforschenonline.org/journals/vaccines/article-data/IJVI-1-104/IJVI-1-104.pdf
Background: WHO and UNICEF supported Ghana to conduct Effective Vaccine Management (EVM) assessments in September 2010 and October 2014 respectively with the view to strengthening vaccine and logistics management for immunization at all levels in the country. In both assessments, temperature monitoring, which is a critical indicator for vaccine potency and immunization programme effectiveness, got very low scores of 50% (2010) and 67% (2014) respectively at the national level and even lower at the regional level. The low scores of temperature monitoring at the national and regional levels in both assessments generated concern and interest for review of the results of the two assessments to determine some of the contributing factors considering Ghana’s advancement in immunization with support of the Gavi Alliance and also with twelve antigens in the routine immunization programme. Objective: The main objective of the review was to determine causes of the poor performance of the temperature monitoring and to devise strategies to improve on it. Methods: Desk review of relevant documents including 2010 and 2014 EVM assessment reports, the report of the 2012 comprehensive EPI Review and the status of implementation of recommendations of the 2010 EVM assessment and the 2012 EPI Review was conducted. Results: The documents review revealed that the 2010 EVM assessment and the 2012 EPI Review recommended installation of continuous electronic temperature monitoring devices in the national and regional vaccine stores but it has not been implemented. The same recommendation has again been made from the 2014 EVM assessment. Conclusions: There was no significant improvement in the scores of temperature monitoring between the 2010 (50%) and 2014 (67%) assessment because of the failure of the Immunization Programme to implement the recommendations of the previous assessment (2010) and the EPI Review (2012) which called for installation of the continuous electronic temperature monitoring devices in the national and regional vaccine stores. Not until the continuous electronic temperature monitoring devices are installed, the scores of temperature monitoring will forever remain low in any future assessment. The question is why use limited resources to conduct EVM assessment if the recommendations to address key challenges of the immunization programme are not to be implemented? Countries conducting EVM in future should make plans to implement the recommendations from the outcome. This is the only way there can be improvement in the immunization programme. Periodic EVM assessment is essential to keep the supply chain system at the required standard. Keywords: Effective Vaccine Management (EVM); EPI; Temperature monitoring; Ghana
VVM’s and MKT Thermometers
Below are some examples of how an MKT thermometer could be used to to simpllfy temperature data analysis for the cold chain.
If the temperature in a refrigerator is 4 deg C for 20 hours then increases to 15 deg C for 4 hours, how will the vaccine life be affected?
- If this is a one day occurrence – should an alarm be set off to notify a repair person?
- If this temperature variation occurs everyday for a month – will it be problematic?
- If on a 24 day excursion the storage temperature is 4 deg C for 20 days and on the last the 4 days the temperature strays to 15 deg C, how would the vaccines be affected?
- If a manufacture builds a SDD refrigerator where the day-time storage temperature is 4 deg and at night the storage temperature increases to 15 deg C, would this be problematic? If this results in considerable cost savings, is this design a good idea?
Graphing data points of this temperature variation, 20 points at 4 deg C and 4 points at 15 deg C, how would you know if this excursion was a problem? Calculating the MKT temperature or using a direct reading MKT thermometer could easily answer these questions. For the example above, the MKT will be below 8 deg C so the temperature excursions to 15 deg C will have a negligible effect on vaccine life.
A direct reading MKT thermometer could be built which would give a continuously updated reading for the past 24 hours. This model could be used to notify a health worker that there is a problem with the equipment. The MKT thermometer would eliminate false alarms. A direct reading MKT thermometer could also look at the temperatures of the past 30 days to evaluate long term performance.
On an excursion a MKT thermometer could be set to start calculating at the beginning of a trip. In the previous example the MKT temperature at the conclusion of the trip the MKT temperature would be below 8 deg C, if the MKT is greater than 8 deg C, the Arrhenius graph such as the one attached could be used to see how the shelf life of the vaccine is affected.
According to Temp Time, a VVM manufacture, “Heat Marker is a patented mean kinetic temperature (MKT) indicator which follows the Arrhenius Equation.” In other words a VVM and an MKT thermometer both are governed by the Arrhenius Equation, one device has a numerical readout and the VVM has a readout which varies the brightness of a dot. Accepting the viability of a VVM is equivalent to accepting the viability of an MKT thermometer.
Some PQS tests are already using MKT temperature to measure the effect of temperature excursions. Widening the use of MKT temperature would simplify data analysis and in the long run reduce cold chain costs by incorporating more realistic guide for temperature excursions.
NEW Vaccine Management Handbook Modules available
Announcing two new publications, part of the Vaccine Management Handbook series:
How to monitor temperatures in the vaccine supply chain (WHO/IVB/15.04)
http://www.who.int/immunization/documents/financing/who_ivb_15.04
How to use passive containers and coolant-packs for vaccine transport and outreach operations (WHO/IVB/15.03)
http://www.who.int/immunization/documents/financing/who_ivb_15.03
Temperature Display
Dear Colleagues,
Many of the prequalified refrigerators have integrated digital thermometers with digital display outside showing instantaneous temperature readings.
During the field visits, I have started to notice considerable number of refrigerators with non-functioning digital display. I am curious if this is a coincidence or commonly observed problem. I would be happy to hear your experiences and suggestions.
Thanks,
Murat
Systematic Remote Temperature Monitoring: Are We Ready?
Systematic Routine Remote Temperature monitoring: are we ready?
Minutes and documents from the Remote Temperature Monitoring workshop, Zanzibar, 4 – 6 November 2014
Remote Temperature Monitoring Technology is recommended to be used at Central Store locations and has been deployed there for the last years. However, data from the EVM Assessments show that this technology is not very widely adopted. Furthermore, as the technology is evolving rapidly and new innovations suggest that the technology is becoming more affordable in the coming years, we asked ourselves in the workshop a few important questions:
Is the current technology “the best we can do”? Is it really meeting the expectations and user needs?
Is there a role for using remote temperature monitoring devices to monitor fridges at sub-national levels? What would a target product profile look like?
What infrastructure, data and management systems need to be in place to ensure the benefits of this innovation can be reaped?
What guidance and tools need to be in place to support governments to make evidence based decisions about which type of technology to adapt?
What data standards need to be put in place to ensure convergence / interoperability of different management systems?
What have we learned about implementation to date? What best practices can be summarized?
23 participants from CHAI, MOH Zanzibar, Nexleaf, PATH, Technical University of Denmark, UNICEF, Village Reach, WHO and the Ministry of Health of Tanzania / Zanzibar participated in a workshop over three days in Zanzibar.
Please find the meeting minutes and key presentations attached.
Selective Highlights:
Currently only two products are WHO pre-qualified, but a detailed landscape analysis from UNICEF suggests that there are many more products available that could potentially be PQS pre-qualified.
Regarding implementation: there is only limited documentation on products, rollout procedures and handover procedures available. Even though the installation of the devices is relatively simple, many challenges arise with the calibration and activation of alarms and human responses to those alarms.
The group clarified that an investment in remote temperature monitoring technology needs to be accompanied by a strong investment into developing data systems (to capture and analyze the data from the devices) and management systems (i.e. SOPs, clarification of roles and responsibilities, capacity building,…).
The work of the working groups will be ongoing aiming to provide a comprehensive documentation package/practical guidance on remote temperature monitoring that will be shared before the Technet meeting of May 2015.
"Cold chain logistics in healthcare - a modern alternative"
An interesting article from pharma-iq.com on RFID and real time monitoring to deliver temperature sensitive drugs. Comments? http://www.pharma-iq.com/cold-chain/articles/cold-chain-logistics-in-healthcare-a-modern-altern/&mac=PharmaIQ_OI_Featured_2011&utm_source=pharma-iq.com&utm_medium=email&utm_campaign=PharmaOptIn&utm_content=7/14/11
- Page :
- 1
- 2