The burden of proof -- refocusing on documented evidence of vaccination history in coverage surveys

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

proof / prüf / (noun)

1. evidence or argument establishing or helping to establish a fact or the truth of a statement

synonyms: evidence, verification, corroboration, authentication, confirmation, certification, documentation, validation, attestation, substantiation

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

When you pick-up and read a vaccination coverage survey report, what do you look at first? Be honest. You skip the background and methods sections of the report and go straight away to the table that details the vaccination coverage. More specifically, your eyes jump to the estimated coverage value for the third dose of diphtheria–tetanus–pertussis containing vaccine (DTPCV3), because, again we are being honest here, that is the door behind which the investment money is hidden.

And when you get to the DTPCV3 column, you have three estimates to choose from: coverage based on documented evidence in home-based records (HBRs) (e.g, vaccination cards, child health books, etc), that based on caregiver recall (relied upon when documented evidence in HBRs is not available) and that based on the combination of both sources. It might look something like this:

Reading the table above, you might then get excited (depending on the chair you sit in) because the estimated DTPCV3 coverage based on the combination of documented evidence and caregiver recall is 90%. That is quite good.

But what if I told you that everything was going to change and the new number to focus on in the table above is not the 90% DTPCV3 coverage by either source of information, but rather the new number to focus on is two lines above in the same column — the DTPCV3 coverage based on documented evidence in HBRs, which in this case is 71%. Wow! There is a big difference in the country above between the DTPCV3 coverage based on both sources and that based on actual documented evidence observed in the HBRs by the survey field teams, 90% vs 71%. As a programme manager or as a supporter of this programme, my decisions about priorities are very different in this case if I am guided by coverage based on documented evidence.

I will put money on the fact that some are asking, “why would we focus on the coverage by documented evidence alone? The 71% DTPCV3 coverage only reflects the contribution to the national coverage estimate based on documented evidence from the 75% of children in the survey above that had a HBR at the time of the survey.” Others may be asking which number is true, 90% or 70%...sorry truth is illusive and something that we do not, and will not, know. But for the same reasons that immigration officers require documented proof of identity using a valid passport on entry to a country or that national health officers at the airports and border entry points in some countries require documented proof of vaccination against polio and/or yellow fever, immunization programmes might hold themselves to account and place greater emphasis on documented proof in HBRs (and facility-based records) of who receives which vaccine, how many doses and when those vaccines were received.

Think about the recommended vaccines for infants in your country. In many countries, the programme recommends infants receive one dose of bacille Calmette-Guérin (BCG) vaccine, a birth dose of polio and hepatitis B vaccines, three doses of diphtheria–tetanus–pertussis containing vaccine (DTPCV) delivered in combination with three doses of hepatitis B (HepB) and Haemophilus influenzae type B (Hib) vaccine, three doses of oral polio vaccine (OPV), one dose of inactivated polio vaccine (IPV), three doses of pneumococcal conjugate vaccine (PCV), two doses of rotavirus (Rota) vaccine, one dose of yellow fever vaccine and one dose of measles containing vaccine (MCV). I will then highlight for you that current research suggests very mixed results of how accurately caregivers can recall their child’s vaccination history. In the 1980s and early 1990s, recalling your child’s vaccination history with some level of accuracy was perhaps possible (there were only five vaccines), but that is less likely today. I know I have a hard time recalling the vaccination histories of my two children.

From a measurement perspective, this is a problem for vaccination coverage surveys which are being relied upon (for better or worse) in many countries as a key mechanism for monitoring programme performance because of poor routine administrative vaccination coverage recording and reporting systems.A review of the most recent DHS and MICS results for documented evidence of vaccination in home-based records and the estimated vaccination coverage among children aged 12-23 months at the time of the survey in 32 Gavi-eligible countries in the WHO African Region suggests that documented evidence of vaccination history was available in only about HALF (!) of the estimated 31 million children born across the 32 countries during 2015 (Figure below), assuming the recent survey results by-and-large reflect the situations in the countries during 2015. Current HBR ownership ranged from around 30% (Chad) to around 95% (Rwanda) with a median value of 70% (IQR: 56-81) (Table below). Ten countries, accounting for just under one-quarter of total births across the 32 countries, were observed to have current HBR ownership levels >80%, with Gambia, Sao Tome and Principe and Rwanda having levels of 90% or greater. Twelve countries had current HBR ownership levels <60%. In Democratic Republic of Congo, Ethiopia and Nigeria—three countries collectively accounting for a little more than 40% of the total 2015 birth cohort across these 32 African countries—a random visit to a household with a child under two years of age in these three countries would identify documented evidence of vaccination history for less than one-third (!) of children based on most recent survey results.

With the problems alluded to above and an understanding that the contribution of coverage based on caregiver recall is almost certainly biased (although we do not know either the direction or the magnitude of the bias), I hope these figures give you pause.

The thought of moving away from utilization of vaccination history based on caregiver recall in coverage survey towards only accepting documented evidence of vaccination history, either from home- or facility-based records, might serve as a wake-up call to those communities where poor recording and reporting in home-based records and facility-based records continues. Over the long-run, this would be an important shift. Is such a shift likely to happen? Perhaps not, although those who conduct nationally representative population-based surveys would love to throw out the vaccination recall section of the survey questionnaire.

A more likely scenario is that over the coming 3-5 years the financial and technical support partners of immunization programmes, perhaps even the national immunization programmes themselves, will agree that good clinical and public health practice benefits from good documentation reflecting the importance of complete, timely, and accurate recording of information. And in doing so, these same partners will begin to take greater notice of the proportion of evidence that is derived from documented evidence in home-based records and facility-based records in vaccination coverage surveys. They will agree that current inefficiencies (i.e., missed opportunities for immunization, extra-immunization) that result from suboptimal documentation are no longer acceptable and that a more effective and efficient system is imperative. They will accelerate investments to understand and correct immunization information system deficiencies in ways not yet considered since continuing on the current path is insufficient and wasteful. And, they will agree that too much that needs to be known remains unknown, and that the poor quality data that exist in many areas of the world fundamentally shapes what we know about the performance of immunization programmes which in turn shapes how decisions are made. And this should change.