Evaluators to assess the MyChild Solution - Gambia

Shifo Foundation, a not for profit social enterprise based in Sweden, develops and implements solutions and methods to strengthen health information systems and child health service delivery. Shifo focuses on ensuring children receive all essential health services, and work to ensure that every child is registered and followed up to receive life-saving health services. Shifo is currently working on the implementation of MyChild Solution, an innovative paper solution with a digital component based on Smart Paper TechnologyTM, that allows for beneficiaries' health data to be captured on paper by health workers and subsequently scanned and digitised.

Context


In 2016, MoHSW signed a Memorandum of Understanding together with ActionAid International The Gambia and Shifo to establish a new form of Health Management Information System (HMIS) in The Gambia. The intervention is implementing the MyChild Solution, a hybrid paper-digital solution that collects, shares, stores and distributes data for the Expanded Programme for Immunisation (EPI) using Smart Paper Technology. The data is digitised and stored in an electronic register from which HMIS reports are generated and shared on health facility, regional and national level. The key stakeholders in the implementation project are Ministry of Health and Social Welfare The Gambia (MoHSW), ActionAid International The Gambia (AAITG), The Vaccine Alliance (GAVI), IKARE, Swedish Postcode Foundation (PKS), af Jochnick Foundation (af Jochnick) and Shifo Foundation (Shifo).


An external evaluation of data quality and operating costs of the MyChild Solution in The Gambia was conducted in 2018 when the solution was being used in 4 fixed and 14 outreach health service delivery points in Western Region 1 and Western Region 2. Based on the findings, the external evaluators recommended the scale up of the solution in the two regions, which was subsequently agreed on by MoHSW, AAITG and Shifo. In July 2018, the scale up of MyChild Solution was commenced in Western Regions 1 and 2 and the solution was scaled up in 20 fixed and 48 outreach health service delivery points.


The key stakeholders are seeking evaluators to assess the MyChild Solution in order to inform the readiness of the solution to be scaled up in other regions in The Gambia. The evaluation is divided into the following components outlined below, each with its own Terms of Reference. Ideally all components should be evaluated by the same evaluators.


Terms of Reference for Component 1: Data Quality

Aim and scope

The purpose of the evaluation is to provide a representation of the level of data quality produced by the MyChild Solution and a comparison of data quality between MyChild Solution and the current HMIS. The evaluation will focus on data collected and processed after the scale-up in Western Region 1 and 2, where the primary provider of capacity building, support and supervision has been the existing health system. The results will give an indication of the quality of data that can be expected after a national scale-up.


The following data quality dimensions will be assessed as part of the evaluation: completeness, timeliness, internal and external consistency and accuracy. The first three dimensions are aligned with the Data Quality Review (DQR) framework and toolkit developed by WHO, The Global Fund, Gavi, The Vaccine Alliance and United States Agency for International Development/MEASURE Evaluation (1). The toolkit is designed to support routine assessments of facility-reported data (WHO, 2017). The fourth dimension, Data Accuracy, is included in the Performance Framework (OI 1.4) for the project (2). The quality of data collected during fixed and outreach sessions will also be compared. The evaluation should also outline the quality assurance mechanisms and time required to verify the data and correct errors in the recognition system.

Research methodology
The evaluators will conduct a statistical analysis on data completeness, timeliness and consistency based on data recorded in the MyChild Solution. Shifo will provide the evaluators access to necessary data and documents in electronic format. 


Data accuracy will be investigated through fieldwork conducted at randomly selected facilities where the MyChild Solution has been used for a minimum of five months, during which most of the workload of registering all children has been completed and health workers have become accustomed to the new system. Data will be collected in conjunction with exit interviews (Component 6) with caregivers after they have attended an immunisation session. Caregivers will be asked for consent to take a picture of the Infant Welfare Card pages containing the child's ID number and vaccines administered. Vaccines administered during the session where data collection was conducted will be compared to vaccines recorded in the electronic immunisation register. Fieldwork will also be required for collecting information on data quality between MyChild Solution and previously used HMIS tools.


A comparison of data quality between the MyChild Solution and the current HMIS system will be conducted by comparing data consistency before and after the implementation of MyChild Solution in a sample of randomly selected facilities where previously used paper-based HMIS tools are still available. Data consistency will be investigated by looking at the level of agreement between the immunisation register, tally sheets, monthly reports and DHIS-2 in MyChild Solution and in the paper-based registers used before the implementation of MyChild Solution.


Assessment of the data quality assurance mechanisms related to data verification and error correction will be conducted through information and statistics obtained from program logs from the data recognition software.

Research questions 

    1. What is the level of data completeness of immunisation sessions captured in electronic reports?
    2. What is the level of timeliness for the submission of monthly HMIS reports?
    3. What is the level of consistency between vaccines recorded in Smart Paper Forms and in electronic data?
    4. Is there any difference in the data quality indicators compared to the pilot evaluation?
    5. Is there any difference in the data quality indicators compared to the existing system?
    6. Quantify the frequency of manual intervention and amount of time needed for correcting the errors in the Smart Papers as part of the quality assurance mechanism?
    7. Is there any difference in the data quality indicators between data collected at fixed and outreach sessions?
    8. What is the level of data accuracy between vaccines recorded in Infant Welfare Card and vaccines recorded in electronic immunisation register?
    9. Have the data quality indicators specified in the Performance Framework met the targets defined?


Terms of Reference for Component 2: Data Use

Aim and scope

The evaluation will focus on assessing the Data for Action intervention in terms of uptake, relevance, understanding and feedback from health workers. The objectives of the proposed research are to map out existing data sources and tools and their use at facility-level, to uncover unmet data needs, to identify programmatic improvements, to evaluate the current Data for Action framework and to provide recommendations for improving Data for Action at the health facility level. The findings will serve as a foundation for building Data for Action at all levels of the healthcare system and establishing effective feedback loops between the levels.

Research methodology

The evaluation will be based on semi-structured interviews with healthcare workers and key stakeholders and observations conducted at randomly selected facilities using MyChild Solution. The interviews and observations will focus on mapping the current tools and data use processes, uncovering unmet data needs and identifying any programmatic changes or improvements. An observation checklist will be used to measure to what extent the different components of the current Data for Action solution such as plotting KPIs on the graphs and reporting on actions taken to improve KPIs are implemented by facilities.

Research questions
  1. What are the tools and data sources available to health workers in facilities using the MyChild Solution and how are they used?
  2. Have the changes in data access and availability with MyChild Solution impacted data use and programmatic activities?
  3. Are there unmet data access needs at facility level?
  4. To what extent are the different components of Data for Action currently implemented by facilities?
  5. What are healthcare workers' and other key stakeholders' perceptions of and feedback on the Data for Action solution?
  6. Is there evidence of any programmatic improvements as a result of changes in  data availability and use with the MyChild Solution?
How can the current Data for Action solution be improved for healthcare facilities?


Terms of Reference for Component 3: Administrative time efficiency

Aim and scope
The objectives of the evaluation is to measure time efficiency for health workers on daily and monthly data administration using MyChild Solution compared to current HMIS tools and to identify how potential time saved on administration has been utilised. Data administration is defined as activities where data related to routine immunization and supplementation services is recorded either using all relevant HMIS tools or the tools in MyChild Solution. The activities cover registering children and updating registration information, recording vaccines and supplementations administered, and daily and monthly administration tasks such as recording vaccine and other stock movements, aggregating beneficiaries per service in daily tally books and preparing monthly returns and vaccine management reports.


Administrative time efficiency is one of the indicators (OI 1.9) in the Results Framework (2) with the target of reducing administrative time by 30% by April 2020 by and an additional 30% by April 2022. The pilot evaluation from 2018 (3) showed that an administrative time saving of 60% had been reached. This evaluation serves as a comparison of administrative time effiency between pilot facilities that received more frequent capacity building and support visits compared to facilities trained during the scale-up by National and Regional EPI Team. The results will give an indication of the time efficiency that can be expected during national scale-up when capacity building and supervision of facilities is conducted primarily by the existing health system structure. The evaluation should also identify how possible time savings have been utilised at facilities.

Research methodology
The evaluation will be based on time-motion observations conducted at randomly selected facilities where the MyChild Solution has been in use for a minimum of five months, during which health workers have become accustomed to the new system. The data for time spent on administration with routine HMIS tools that was published in the pilot evaluation in 2018 will be used as a comparison. In addition, time efficiency published for pilot facilities in the previous evaluation will be compared to time spent at facilities trained during regional scale-up. In order to identify how potential administrative time saved is being utilised and whether the time savings are linked to activities that provide better quality of care for beneficiaries, qualitative interviews will be conducted with health workers.

Research questions
  1. What is the time spent on daily and monthly administration per one fully immunised child with MyChild Solution compared to using all routine HMIS tools required for EPI data collection?
  2. Is there a difference in time efficiency between pilot facilities and facilities trained during regional scale-up?
  3. Has the target for administrative time savings in the Performance Framework been met?
  4. How are health workers utilising the potential time saved on data administration?


Terms of Reference for Component 4: Sustainability evaluation of MyChild Solution – Total cost of ownership and gradual transition of work processes

Aim and scope

The sustainability evaluation will assess the total cost of ownership for MyChild Solution and gradual transition of work processes. The analysis of the total cost of ownership will compare the annual operating costs of MyChild Solution with the annual operating costs of the current HMIS for preventive child health program and inform MoHSW on the required operating costs to sustain MyChild Solution on a national level. The second aim of the evaluation is to measure the extent to which MyChild Solution work processes have been transferred at the time of the evaluation and to assess the sustainability of each work process when integrated into the existing health system. 

 

Research methodology

Analysis of the total cost of ownership will compare the annual operating costs of MyChild Solution with the annual operating costs of the current HMIS for preventive child health program (vaccination and supplementation). The evaluation will include costs for printing, equipment, data verification and quality assurance processes (including the HR costs), trainings related to data quality and use of the tools for both the current HMIS and the MyChild Solution, costs of Smart Paper Technology Engine maintenance and development and other relevant costs. The current funder for each cost item in the current HMIS will be identified. Cost analysis will be done separately with and without administrative time-savings. Additional data needed to estimate the costs of MyChild Solution will be obtained in consultation with project team implementing the system in Western Regions 1 and 2 as well as MoHSW in The Gambia.  

The evaluation of transfer of work processes will be based on observations and semi-structured interviews conducted at facility and regional level. A set of criteria to evaluate the level and success of the transfer and the sustainability of each work process will be developed by evaluators based on defined Standard Operating Procedures (SOP). The assessment of the sustainability of the work processes should investigate whether work processes are performed according to SOPs and whether persons responsible for the work processes have sufficient knowledge to transfer the work process to another person. An assessment of the sustainability of the work processes that remain to be transferred will also be conducted based on a set of criteria to be developed by evaluators based on existing standards and criteria where possible. The suitability of MyChild Solution to be used during immunization campaigns will also be assessed.  


Research questions

      1. Which work processes have been integrated into the existing health system structure (in particular, integration with DHIS2) at the time of the evaluation and what proportion remain to be integrated?
      2. What is the level of transfer success and sustainability of each work process that has been integrated?
      3. What is the total cost of ownership for MyChild Solution compared to the current HMIS (based on all cost elements mentioned above)?
      4. What is the suitability of MyChild Solution to support immunization campaigns?


Terms of Reference for Component 5: Perceptions and experiences of My Child Solution

Aim and scope

Investigate the perceptions and experiences of caregivers and users along the healthcare chain from facility to national level on the use of MyChild Solution. The evaluation will provide recommendations on improvements for the MyChild Solution from the perspective of both users and beneficiaries.  


Research methodology

Semi-structured interviews and possible focus group discussions will be conducted at randomly selected facilities where the MyChild Solution has been used for a minimum of five months, covering both the health workers' experiences of the intial transition and implementation as well as the routine use of the solution. Interviews will be conducted at all levels of the health care structure: with caregivers, with health workers, with regional health directorate staff and with national EPI staff. Semi-structured exit interviews will be sought with caregivers attending immunisation sessions who report having attended the clinic in the past when routine HMIS tools were used and report having received at least one SMS reminder. Thematic analysis will be used to uncover recurring themes in the interviews.  


Research questions
    • What are the perceptions of health facility workers, regional health directorate staff and national EPI on the MyChild Solution?
    • What are the perceptions of caregivers on MyChild Solution in general and specifically on the following components:
      1. Usefulness of SMS reminders
      2. Changes to health care service delivery such as waiting times when comparing the experience of the current visit (with MyChild Solution) compared to past experiences (with HMIS tools)

Timeline
  • The evaluation is set to start in March 2019
  • A detailed timeline and deadlines for the deliverables will be developed and agreed on in consultation with the selected evaluators


Conditions applying to all Terms of Reference

Deliverables
  1. Detailed timeline and methodology
  2. One or several draft reports (number based on amount of revisions required)
  3. Final report

Timeline T
The evaluation is set to start in March 2019 and expected to be finalized by the end of May. A detailed timeline and deadlines for the deliverables will be developed and agreed on in consultation with the selected evaluators.

Collaboration 
The evaluator(s) will work closely with MoHSW, AAITG and Shifo Foundation. With the exclusion of fieldwork conducted in The Gambia, the evaluator(s) can work remotely from their current location. For obtaining and verifying data sources, the evaluator(s) will receive the aggregated data in csv format for data analysis and relevant documents in electronic format. The key stakeholders are expected to be notified immediately of any concerns or delays that may arise during the course of the engagement.


Proposal submission requirements

Interested parties are invited to submit their CV and a technical proposal (in Word or PDF format) to: MoHSW The Gambia EPI Secretariat This email address is being protected from spambots. You need JavaScript enabled to view it., Secretariat of Shifo Foundation, This email address is being protected from spambots. You need JavaScript enabled to view it. and Secretariat of AAITG, This email address is being protected from spambots. You need JavaScript enabled to view it.

In the technical proposal, include the evaluation methodology to be used, relevant experience based on evaluation criteria, and a detailed timeline of activities.

Proposals must be received no later than 18th of February 2019. You are solely responsible for ensuring that your proposal is delivered on time. All costs incurred in preparing proposals will be borne by the applicant(s).


Proposal evaluation criteria

The evaluator(s) will be selected by selection committee, consisting of MoHSW EPI Manager, AAITG Project Coordinator, Shifo Project Coordinator, GAVI secratariat, IKARE Project Coordinator, PKS Project Coordinator and af Jochnick Project Coordinator. The selection will be based on the basis of the following criteria:

    • Proven ability to perform high-quality independent research and evaluation, preferably at PhD level or higher
    • Practical experience in human centred design and developing health innovations, preferably for low-resource settings
    • Experience and understanding of image processing and extraction of quantitative data from images
    • Practical experience of delivering immunisation services in African continent, preferably in The Gambia
    • Practical experience in immunisation program management in African continent, preferably in The Gambia
    • Good understanding of health workers' needs in data during and after service delivery, preferably in immunisation domain
    • Good understanding and practical experience of evaluating data quality using WHO DQR Framework, preferably within immunisation domain
    • Good understanding, knowledge of tools and practical experience from analysing various types of data from databases, preferably in R, Python or SAS
    • Good knowledge of causal inference methods in health sciences and experience in evaluating the impact of health interventions, preferably in evaluating the process of decision-making in healthcare
    • Practical experience from doing cost analysis of health interventions, preferably within immunisation domain
Terms of payment 
Payments will be initiated upon successful completion or documented progress made on tasks as mentioned in the above Terms of Reference.


References

    1. World Health Organization. (2017) Data quality review: a toolkit for facility data quality assessment. Module 1. Framework and metrics. Geneva: World Health Organization.
    2. Performance Framework – Every Child Counts The Gambia - https://doc.shifo.org/display/shareit/32408577/YWW693c676fbbe348a88d2ec33f109d8d56IGL
    3. Assessment of MyChild Solution in The Gambia: Data quality, administrative time efficiency, operation costs, and users' experiences and perceptions -  https://shifo.org/doc/rmnch/MyChildExternalEvaluationReportGambia2018.pdf

Job vacancy

  •   Monday, 18 February 2019
  • Stockholm, Sweden