In 2016, MoHSW signed a Memorandum of Understanding together with ActionAid International The Gambia and Shifo to establish a new form of Health Management Information System (HMIS) in The Gambia. The intervention is implementing the MyChild Solution, a hybrid paper-digital solution that collects, shares, stores and distributes data for the Expanded Programme for Immunisation (EPI) using Smart Paper Technology. The data is digitised and stored in an electronic register from which HMIS reports are generated and shared on health facility, regional and national level. The key stakeholders in the implementation project are Ministry of Health and Social Welfare The Gambia (MoHSW), ActionAid International The Gambia (AAITG), The Vaccine Alliance (GAVI), IKARE, Swedish Postcode Foundation (PKS), af Jochnick Foundation (af Jochnick) and Shifo Foundation (Shifo).
An external evaluation of data quality and operating costs of the MyChild Solution in The Gambia was conducted in 2018 when the solution was being used in 4 fixed and 14 outreach health service delivery points in Western Region 1 and Western Region 2. Based on the findings, the external evaluators recommended the scale up of the solution in the two regions, which was subsequently agreed on by MoHSW, AAITG and Shifo. In July 2018, the scale up of MyChild Solution was commenced in Western Regions 1 and 2 and the solution was scaled up in 20 fixed and 48 outreach health service delivery points.
The key stakeholders are seeking evaluators to assess the MyChild Solution in order to inform the readiness of the solution to be scaled up in other regions in The Gambia. The evaluation is divided into the following components outlined below, each with its own Terms of Reference. Ideally all components should be evaluated by the same evaluators.
Terms of Reference for Component 1: Data Quality
Aim and scope
The purpose of the evaluation is to provide a representation of the level of data quality produced by the MyChild Solution and a comparison of data quality between MyChild Solution and the current HMIS. The evaluation will focus on data collected and processed after the scale-up in Western Region 1 and 2, where the primary provider of capacity building, support and supervision has been the existing health system. The results will give an indication of the quality of data that can be expected after a national scale-up.
The following data quality
dimensions will be assessed as part of the evaluation: completeness,
timeliness, internal and external consistency and accuracy. The first three
dimensions are aligned with the Data Quality Review (DQR) framework and
toolkit developed by WHO, The Global Fund, Gavi, The Vaccine Alliance and
United States Agency for International Development/MEASURE Evaluation (1). The
toolkit is designed to support routine assessments of
facility-reported data (WHO, 2017). The fourth dimension, Data Accuracy, is
included in the Performance Framework (OI 1.4) for the
project (2). The quality of data collected during fixed and outreach sessions will
also be compared. The evaluation should also outline the quality assurance
mechanisms and time required to verify the data and correct errors in the
The evaluators will conduct a statistical analysis on data completeness, timeliness and consistency based on data recorded in the MyChild Solution. Shifo will provide the evaluators access to necessary data and documents in electronic format.
Data accuracy will be
investigated through fieldwork conducted at randomly selected facilities where
the MyChild Solution has been used for a minimum of five months, during which
most of the workload of registering all children has been completed and health
workers have become accustomed to the new system. Data will be collected in
conjunction with exit interviews (Component 6) with caregivers after
they have attended an immunisation session. Caregivers will be asked for
consent to take a picture of the Infant Welfare Card pages containing the
child's ID number and vaccines administered. Vaccines administered during the
session where data collection was conducted will be compared to vaccines
recorded in the electronic immunisation register. Fieldwork will also be
required for collecting information on data quality between MyChild Solution
and previously used HMIS tools.
A comparison of data quality between the MyChild Solution and the
current HMIS system will be conducted by comparing data consistency before and
after the implementation of MyChild Solution in a sample of randomly selected
facilities where previously used paper-based HMIS tools are still available.
Data consistency will be investigated by looking at the level of agreement
between the immunisation register, tally sheets, monthly reports and DHIS-2 in
MyChild Solution and in the paper-based registers used before the
implementation of MyChild Solution.
Assessment of the data quality assurance mechanisms related to data
verification and error correction will be conducted through information and
statistics obtained from program logs from the data recognition software.
Terms of Reference for Component 2: Data Use
Aim and scope
The evaluation will focus on assessing the Data for Action intervention in terms of uptake, relevance, understanding and feedback from health workers. The objectives of the proposed research are to map out existing data sources and tools and their use at facility-level, to uncover unmet data needs, to identify programmatic improvements, to evaluate the current Data for Action framework and to provide recommendations for improving Data for Action at the health facility level. The findings will serve as a foundation for building Data for Action at all levels of the healthcare system and establishing effective feedback loops between the levels.
The evaluation will be based on semi-structured interviews with healthcare workers and key stakeholders and observations conducted at randomly selected facilities using MyChild Solution. The interviews and observations will focus on mapping the current tools and data use processes, uncovering unmet data needs and identifying any programmatic changes or improvements. An observation checklist will be used to measure to what extent the different components of the current Data for Action solution such as plotting KPIs on the graphs and reporting on actions taken to improve KPIs are implemented by facilities.
Terms of Reference for Component 3: Administrative time efficiency
Aim and scope
The objectives of the evaluation is to measure time efficiency for health workers on daily and monthly data administration using MyChild Solution compared to current HMIS tools and to identify how potential time saved on administration has been utilised. Data administration is defined as activities where data related to routine immunization and supplementation services is recorded either using all relevant HMIS tools or the tools in MyChild Solution. The activities cover registering children and updating registration information, recording vaccines and supplementations administered, and daily and monthly administration tasks such as recording vaccine and other stock movements, aggregating beneficiaries per service in daily tally books and preparing monthly returns and vaccine management reports.
efficiency is one of the indicators (OI 1.9) in the Results Framework (2)
with the target of reducing administrative time by 30% by April 2020 by and an
additional 30% by April 2022. The pilot evaluation from 2018 (3) showed that an
administrative time saving of 60% had been reached. This evaluation serves as a
comparison of administrative time effiency between pilot facilities that
received more frequent capacity building and support visits compared to
facilities trained during the scale-up by National and Regional EPI Team. The
results will give an indication of the time efficiency that can be expected
during national scale-up when capacity building and supervision of facilities
is conducted primarily by the existing health system structure. The
evaluation should also identify how possible time savings have been utilised at
The evaluation will be based on time-motion observations conducted at randomly selected facilities where the MyChild Solution has been in use for a minimum of five months, during which health workers have become accustomed to the new system. The data for time spent on administration with routine HMIS tools that was published in the pilot evaluation in 2018 will be used as a comparison. In addition, time efficiency published for pilot facilities in the previous evaluation will be compared to time spent at facilities trained during regional scale-up. In order to identify how potential administrative time saved is being utilised and whether the time savings are linked to activities that provide better quality of care for beneficiaries, qualitative interviews will be conducted with health workers.
Terms of Reference for Component 4: Sustainability evaluation of MyChild Solution – Total cost of ownership and gradual transition of work processes
Aim and scope
The sustainability evaluation will assess the total cost of ownership for MyChild Solution and gradual transition of work processes. The analysis of the total cost of ownership will compare the annual operating costs of MyChild Solution with the annual operating costs of the current HMIS for preventive child health program and inform MoHSW on the required operating costs to sustain MyChild Solution on a national level. The second aim of the evaluation is to measure the extent to which MyChild Solution work processes have been transferred at the time of the evaluation and to assess the sustainability of each work process when integrated into the existing health system.
Analysis of the total cost of ownership will compare the annual
operating costs of MyChild Solution with the annual operating costs of the
current HMIS for preventive child health program (vaccination and
supplementation). The evaluation will include costs for printing, equipment,
data verification and quality assurance processes (including the HR costs), trainings
related to data quality and use of the tools for both the current HMIS and the
MyChild Solution, costs of Smart Paper Technology Engine maintenance and
development and other relevant costs. The current funder for each cost item in
the current HMIS will be identified. Cost analysis will be done separately with
and without administrative time-savings. Additional data needed to estimate the
costs of MyChild Solution will be obtained in consultation with project team
implementing the system in Western Regions 1 and 2 as well as MoHSW in The
The evaluation of transfer
of work processes will be based on observations and semi-structured interviews
conducted at facility and regional level. A set of criteria to evaluate the level and success of the transfer
and the sustainability of each work process will be developed by
evaluators based on defined Standard Operating Procedures (SOP). The assessment
of the sustainability of the work processes should investigate whether work
processes are performed according to SOPs and whether persons responsible for the
work processes have sufficient knowledge to transfer the work process to
another person. An assessment of
the sustainability of the work processes that remain to be transferred will
also be conducted based on a set of criteria to be developed by evaluators
based on existing standards and criteria where possible. The suitability of
MyChild Solution to be used during immunization campaigns will also be
Terms of Reference for Component 5: Perceptions and experiences of My Child Solution
Aim and scope
Investigate the perceptions and experiences of caregivers and users
along the healthcare chain from facility to national level on the use of
MyChild Solution. The
evaluation will provide recommendations on improvements for the MyChild
Solution from the perspective of both users and beneficiaries.
and possible focus group discussions will be conducted at randomly selected
facilities where the MyChild Solution has been used for a minimum of five
months, covering both the health workers' experiences of the intial transition
and implementation as well as the routine use of the solution. Interviews
will be conducted at all levels of the health care structure: with caregivers,
with health workers, with regional health directorate staff and with national
EPI staff. Semi-structured exit interviews will be sought with
caregivers attending immunisation sessions who report having attended the
clinic in the past when routine HMIS tools were used and report having received
at least one SMS reminder. Thematic analysis will be used to uncover recurring
themes in the interviews.
- Detailed timeline and methodology
- One or several draft reports (number based on amount of revisions required)
- Final report
Proposal submission requirements
In the technical proposal, include the evaluation methodology to be used, relevant experience based on evaluation criteria, and a detailed timeline of activities.
Proposals must be received no later than 18th of February 2019. You are solely responsible for ensuring that your proposal is delivered on time. All costs incurred in preparing proposals will be borne by the applicant(s).
Proposal evaluation criteriaThe evaluator(s) will be selected by selection committee, consisting of MoHSW EPI Manager, AAITG Project Coordinator, Shifo Project Coordinator, GAVI secratariat, IKARE Project Coordinator, PKS Project Coordinator and af Jochnick Project Coordinator. The selection will be based on the basis of the following criteria:
- Proven ability to perform high-quality independent research and evaluation, preferably at PhD level or higher
- Practical experience in human centred design and developing health innovations, preferably for low-resource settings
- Experience and understanding of image processing and extraction of quantitative data from images
- Practical experience of delivering immunisation services in African continent, preferably in The Gambia
- Practical experience in immunisation program management in African continent, preferably in The Gambia
- Good understanding of health workers' needs in data during and after service delivery, preferably in immunisation domain
- Good understanding and practical experience of evaluating data quality using WHO DQR Framework, preferably within immunisation domain
- Good understanding, knowledge of tools and practical experience from analysing various types of data from databases, preferably in R, Python or SAS
- Good knowledge of causal inference methods in health sciences and experience in evaluating the impact of health interventions, preferably in evaluating the process of decision-making in healthcare
- Practical experience from doing cost analysis of health interventions, preferably within immunisation domain
- World Health Organization. (2017) Data quality review: a toolkit for facility data quality assessment. Module 1. Framework and metrics. Geneva: World Health Organization.
- Performance Framework – Every Child Counts The Gambia -
- Assessment of MyChild Solution in The Gambia: Data quality, administrative time efficiency, operation costs, and users' experiences and perceptions -
Monday, 18 February 2019