Advertisement
Review

Measuring Coverage in MNCH: Design, Implementation, and Interpretation Challenges Associated with Tracking Vaccination Coverage Using Household Surveys

  • Felicity T. Cutts mail,

    felicity.cutts@lshtm.ac.uk

    Affiliation: Independent consultant, La Londe les Maures, France

    X
  • Hector S. Izurieta,

    Affiliation: Food and Drug Administration, Rockville, Maryland, United States of America

    X
  • Dale A. Rhoda

    Affiliation: Battelle Memorial Institute, Columbus, Ohio, United States of America

    X
  • Published: May 07, 2013
  • DOI: 10.1371/journal.pmed.1001404
  • Published in PLOS Medicine

Abstract

Vaccination coverage is an important public health indicator that is measured using administrative reports and/or surveys. The measurement of vaccination coverage in low- and middle-income countries using surveys is susceptible to numerous challenges. These challenges include selection bias and information bias, which cannot be solved by increasing the sample size, and the precision of the coverage estimate, which is determined by the survey sample size and sampling method. Selection bias can result from an inaccurate sampling frame or inappropriate field procedures and, since populations likely to be missed in a vaccination coverage survey are also likely to be missed by vaccination teams, most often inflates coverage estimates. Importantly, the large multi-purpose household surveys that are often used to measure vaccination coverage have invested substantial effort to reduce selection bias. Information bias occurs when a child's vaccination status is misclassified due to mistakes on his or her vaccination record, in data transcription, in the way survey questions are presented, or in the guardian's recall of vaccination for children without a written record. There has been substantial reliance on the guardian's recall in recent surveys, and, worryingly, information bias may become more likely in the future as immunization schedules become more complex and variable. Finally, some surveys assess immunity directly using serological assays. Sero-surveys are important for assessing public health risk, but currently are unable to validate coverage estimates directly. To improve vaccination coverage estimates based on surveys, we recommend that recording tools and practices should be improved and that surveys should incorporate best practices for design, implementation, and analysis.

This paper is part of the PLOS Medicine “Measuring Coverage in MNCH” Collection.

Introduction

The percentage of a population that has been vaccinated—vaccination coverage—is an imperfect but helpful measure of the effectiveness of vaccination programs and of public health more broadly [1]. Vaccination coverage is a tracer condition for results-based financing [2], an indicator of eligibility for Millennium Challenge Account assistance [3], and a criterion for support from the GAVI Alliance for the introduction of new vaccines [4]. Making funding decisions contingent on coverage potentially incentivizes inflation of coverage figures, and there is wide recognition of the need to improve the data [5][8].

Ideally, vaccination coverage should be monitored continuously using registries or administrative reports [9]. Electronic immunization registries aim to document all vaccinations of each individual in each birth cohort [10],[11]. Denominators may derive from the same registry [11] or from a separate vital statistics system. When well implemented, electronic immunization registries can provide data for coverage measurement and for program management activities such as monitoring vaccine supply and requisitions and sending vaccination reminders. However, challenges facing such registries include accounting for migration within and between countries, ensuring complete birth registration and vaccination reporting, avoiding record duplication [12], and ensuring continuity after organizational changes [13]. Although pilot studies of electronic registries are ongoing in low- and middle-income countries including Albania, Guatemala, India, and Viet Nam [14], these challenges currently limit their use. Therefore, in most low- and middle-income countries, “administrative coverage” is calculated using aggregate reported data on the number of doses of each vaccine administered to children in the target age group in a given time period and target population estimates from censuses [1]. Health workers at each health facility typically compile data manually each month from clinic records such as immunization registers or tally sheets. At each vaccination visit, the health worker records vaccinations on clinic records and on a vaccination card, child health card, or other home-based record (HBR) that the mother keeps. The HBR serves as an educational tool for the mother and is also an important data source in household surveys. In many countries, however, the quality of primary recording of vaccinations, of transcription and compilation of data, and of reporting is low, and numerators may be either inflated (e.g., because doses outside the recommended age range are included) or too low (e.g., if private practitioners do not report). Moreover, denominators are often grossly inaccurate [5],[7]. Hence, wherever possible other data sources such as surveys are still considered in the World Health Organization (WHO)–United Nations Children's Fund (UNICEF) estimates of national immunization coverage [15].

Given current problems with coverage estimates based on administrative reports in many countries, we believe that surveys will continue to provide important information in the short-to-medium term, at national and sub-national levels. It is therefore critical that surveys are conducted rigorously. In this review, which is part of the PLOS Medicine “Measuring Coverage in MNCH” Collection, we discuss the survey methods used to estimate vaccination coverage in low- and middle-income countries, highlight potential pitfalls, and propose strategies to improve coverage measurement. Our review aims to inform public health practitioners and the researchers who design and implement surveys as well as Ministry of Health officials and donors who interpret and use data from surveys.

Survey Methods Used to Measure Vaccination Coverage

Four types of surveys are commonly employed to estimate vaccination coverage (Table 1). The Demographic and Health Surveys (DHS) [16] and Multiple Indicator Cluster Surveys (MICS) [17] are probability sample surveys, in which each household has a known and nonzero probability of being selected in the sample. There have been about 10–15 DHS and 20 MICS per year since 1995. These large, important, and generally well-conducted household surveys, which are used to collect data about many aspects of health, are described in detail in a companion paper in this Collection [18].

thumbnail

Table 1. Characteristics of common surveys used to measure vaccination.

doi:10.1371/journal.pmed.1001404.t001

The Expanded Programme on Immunization (EPI) cluster survey was developed by the WHO and was described in 1982 as a practical tool to quickly estimate coverage to within ±10 percentage points of the point estimate [19]. The original EPI survey method selects 30 clusters from which seven children in each cluster are selected using the “random start, systematic search” method. Specifically, a starting dwelling is chosen by starting at a central location in the village or town, selecting a direction at random, counting the dwellings lying in that direction up to the edge of the village, and selecting one of them randomly; adjacent households are then visited until seven children aged 12–23 months have been enrolled [20],[21]. The central starting location may bias the method to include households with good access to vaccination, so it is difficult to assign unbiased probabilities of selection to the households using this method, which does not meet the above criteria for a probability sample and is, therefore, a “non-probability sampling” survey method [22]. EPI surveys are widely used at national and sub-national levels, but there is no central database of results, so the total number of surveys conducted is unknown. Adaptations of the EPI survey have incorporated probability sampling at the final stage of sample selection [22][26], and the updated WHO guidelines [21] as well as a recent companion manual on hepatitis B immunization surveys emphasize the need for probability sampling for scientifically robust estimates of coverage [27].

The main design differences between EPI surveys (if probability sampling is used) and DHS or MICS surveys is that EPI surveys focus specifically on vaccination data while DHS and MICS surveys cover a wide range of population and health topics and include a much larger sample size. In addition, field implementation of EPI surveys is variable and often done without external technical assistance, while the DHS and MICS are highly standardized and have substantial technical assistance and quality control.

A final household survey method commonly used to estimate health intervention coverage in low- and middle-income countries is Lot Quality Assurance Sampling (LQAS). LQAS surveys use a stratified sampling approach to classify “lots,” which might be districts, health units, or catchment areas, as having either “adequate” or “inadequate” coverage of various public health interventions. For vaccination coverage measurement, LQAS is “nested” within a cluster survey to evaluate neonatal tetanus elimination [28], coverage of yellow fever vaccination [29], and coverage of meningococcal vaccine campaigns [30], and to monitor polio vaccination coverage after supplementary immunization activities [31].

Survey Design and Implementation

Surveys used to estimate vaccination coverage should have a sample size that results in an acceptable sampling error, and implementation should minimize non-sampling errors, including selection bias and information bias (Table 2). In DHS and MICS surveys, the sample size is determined by the estimated number of households required for the desired precision of key indicators (not vaccination coverage), and all children in the eligible age groups in those households are included. In recent DHS surveys this design has given sample sizes of around 1,800 children aged 12–23 months. EPI surveys traditionally included 210 children aged 12–23 months in 30 clusters, but the sample size and number of clusters should be calculated according to assessments of the likely coverage, intra-cluster correlation, and desired precision of the vaccination coverage estimate [21],[27].

thumbnail

Table 2. Main potential sources of error and strategies to minimize them in population-based surveys measuring vaccination coverage.

doi:10.1371/journal.pmed.1001404.t002

Selection bias may occur due to use of an outdated or nonrepresentative sampling frame, use of non-probability sampling, or poor field worker practices such as substituting a selected household with one that is easier to reach. The “random start, systematic search” method used in traditional EPI surveys has intrinsic geographic bias. It allows field workers to select households rather than this being part of the initial sampling process, does not document reasons for nonparticipation, and cannot adjust for biases resulting from out-of-date size estimates for selection of clusters using probability proportional to estimated size sampling. Moreover, in EPI and LQAS surveys, teams are likely to replace households where no one is home or where eligible respondents refuse to participate. If respondents are not selected randomly and if the same forces that influence participation in the survey also influence participation in vaccination (e.g., families missed by interviewers because they work in the fields all day may also lack time to attend vaccination clinics), replacement is likely to result in bias, probably upwards. Finally, surveys of the vaccination status of living individuals are inherently subject to selection bias since death is more likely in unvaccinated than in vaccinated children. In settings where there is a high infant mortality rate, this bias may be substantial.

There are multiple potential sources of information error and bias in measuring the vaccination status of each child in surveys (Figure 1), many of which also affect data included in administrative reports. Mistakes can occur during primary data recording each time a child attends a vaccination point or when survey interviewers transcribe birth and vaccination dates onto a paper or digital questionnaire. If a paper questionnaire is used, further errors can occur during digital data entry. Data on source documents can also be incomplete [32] or inaccurate [33] and, when new vaccines are introduced, old HBRs may remain in circulation, requiring health workers to improvise in their recording (Figure 2). There is further confusion regarding recording of vaccines administered during campaigns such as “vaccination weeks” on HBRs [34].

thumbnail

Figure 1. Schematic of recording of vaccination data at the time of vaccination and during community surveys.

Recording at the time of vaccination (primary recording) is indicated in black boxes; recording during surveys is indicated in green boxes. Main potential sources of information error and bias are highlighted in blue. DOB, date of birth.

doi:10.1371/journal.pmed.1001404.g001
thumbnail

Figure 2. Several instances of improvisation on a vaccination card.

(Photo courtesy of Carolina Danovaro, Pan American Health Organization.)

doi:10.1371/journal.pmed.1001404.g002

When the HBR is not available (it may be lost or locked away, or mothers may not be given enough time to find it), interviewers question the child's parent or guardian to construct a verbal vaccination history. The reliability of such histories may vary with the information received or understood by mothers at the time of vaccination; the interviewer's skills, carefulness, neutral demeanor, and use of appropriate language; the recall period; and the length of the questionnaire and resulting interview fatigue [35][37]. The complexity of the vaccination schedule can also affect the reliability of a verbal vaccination history. When the EPI survey was introduced in the 1980s, the infant EPI schedule comprised five visits, which lent themselves to straightforward questions to the mother (Table 3). Because current schedules are much more complex (Table 4) and vary over time and between countries, constructing a verbal history of vaccinations received is now much more difficult and likely to become increasingly so. Thus, the questions included in surveys need substantial and continuous adaptation.

thumbnail

Table 3. Illustrative questions used in the past to elicit a verbal history of vaccination according to the EPI schedule in the 1980s.

doi:10.1371/journal.pmed.1001404.t003
thumbnail

Table 4. World Health Organization–recommended EPI schedule, 2012.

doi:10.1371/journal.pmed.1001404.t004

Data Analysis and Reporting Issues

Traditionally, surveys report the proportion of persons who have been vaccinated as recorded “by card only” and by “card plus history,” both by age 12 months and by age at the time of the survey. EPI surveys also calculate and report separately on coverage of “valid” doses among children with cards, such as a minimum interval of 28 days between doses of diphtheria-tetanus-pertussis-containing vaccines and a minimum age of 270 days for measles vaccination [21]. As coverage increases, evaluation of the timeliness of vaccination among children with documented dates of birth, and of each vaccine dose, provides additional information to guide program performance. Timeliness can be illustrated through graphs of the distribution of age at receipt of each dose compared to the national schedule [32],[38] or by time-to-event curves of the cumulative coverage by age [32],[39]. The mean number of extra days or weeks that children remain under-vaccinated and at risk of disease [38],[40],[41] and risk factors for delay in vaccination can be assessed [39],[40].

Surveys that use probability proportional to estimated size sampling without stratification assume that each cluster has equal weight in the analysis. EPI surveys do not collect data on the number of eligible households in each cluster, and cannot validate this assumption. Consequently, if outdated or inaccurate sampling frames are used, the resulting point estimate may be biased. If, however, a household listing step is included in the survey preparation and sampling stages, appropriate weights can be calculated and used to derive national estimates and confidence intervals, as is recommended nowadays by DHS and MICS protocols [18].

The standard error of the coverage estimate is traditionally used to calculate and report a 95% confidence interval around the point estimate. The confidence interval is affected by the sample size, the sampling design, and the underlying proportion itself. Because individuals living in one cluster of a population tend to be more similar to each other than persons from different clusters, respondents in a cluster sample each contribute less independent information about the overall population than respondents in a simple random sample. This positive intra-cluster correlation causes cluster samples to have a wider confidence interval around the point estimate of the population parameter than a simple random sample of the same size. DHS, MICS, and EPI surveys all provide guidance on estimation of confidence intervals for key indicators, but the degree to which confidence intervals are reported and used varies widely, as discussed elsewhere in this Collection [42].

The application of standard statistical techniques to estimate confidence intervals has been challenged for surveys that use non-probability sampling of households within each cluster [43], although simulations of results from EPI surveys have shown that confidence intervals in these surveys are generally within the desired precision of ±10 percentage points [44]. Some variations on the EPI survey method take a probability sample (e.g., a systematic random sample in the final stage) [22][26], which makes it possible to calculate sampling weights and construct meaningful confidence intervals.

LQAS surveys inevitably have a central range of coverage (the gray area) that is not excluded by either the “adequate” or the “inadequate” classification. That is, neither classification excludes the medium category. For fixed values of alpha and beta (the probability of type I and II errors, respectively), a larger sample size per lot will result in a narrower gray area and a correspondingly more confident conclusion about whether coverage is likely to be adequate (Figure 3). When data are combined across numerous lots, it is possible to estimate a region-wide proportion and confidence interval using formulae from stratified sampling and applying strata and cluster weights. However, at the level of the individual lot, the user does not obtain a precise coverage estimate from a LQAS survey, but only an assurance that coverage in populations where there is very low coverage is very likely to be classified as inadequate and that coverage in populations where there is very high coverage is very likely to be classified as adequate.

thumbnail

Figure 3. Operating characteristic curves for four LQAS sampling plans.

In each panel, the curve indicates the probability of finding d* or more vaccinated children in a random sample of size n. Lots with coverage≤lower threshold (LT) will be classified as having inadequate coverage with probability ≥(1 − α). Lots with coverage≥upper threshold (UT) will be classified as having adequate coverage with probability ≥(1 − β). The gray area is the region where LT<coverage<UT; lots with coverage in the gray area may be labeled either adequate or inadequate. The gray area includes the region of coverage, for instance, where there is a 50/50 probability of being classified adequate or inadequate. Neither classification (adequate or inadequate) rules out the strong possibility that the true coverage lies in the gray area. The gray area may be made larger or smaller and may be moved to regions of higher or lower coverage by manipulating LT, UT, α, and β to arrive at different values of n and d*.

doi:10.1371/journal.pmed.1001404.g003

In the analysis phase, survey analyses are usually restricted to respondents with complete data. However, analyses of DHS and MICS surveys prior to 2002 showed that maternally recalled vaccination data were internally consistent, and that inclusion of a verbal history of vaccination in results was preferable to other options such as restricting analyses to children with HBRs or assuming that coverage among those without HBRs was the same as those with HBRs [45]. Inclusion of children without HBRs is only possible for calculation of percentage coverage, however, and not for assessing timeliness of vaccination.

When measuring vaccination coverage from survey data, if data are missing for reasons related to the likelihood of vaccination, restricting analyses to those with complete data will bias results. Analysts can impute hypothetical values for missing responses, but imputing a single value fails to account correctly for the uncertainty associated with selecting an arbitrary (though perhaps plausible) value to impute. More sophisticated methods include integrating over a likelihood function, or imputing numerous values for each missing datapoint (multiple imputation), and are preferred but rare in vaccination coverage measurement [46],[47]. DHS and MICS surveys adjust sampling weights for nonresponse, and may impute a single value when a HBR vaccination date is invalid (e.g., lists day 32 of the month) or missing, but do not currently employ multiple imputation in vaccination coverage estimates [48], although this could be introduced in the future as the technique of multiple imputation becomes more accessible.

How Can the Challenges Facing the Measurement of Vaccination Coverage Be Addressed?

Vaccination coverage is an important indicator that is used to monitor not only immunization programs but also health system performance at national and global levels. Coverage surveys can include questions on reasons for receiving, or not receiving, vaccines, and investigate demographic and other factors associated with coverage [49]. In addition, in the specific context of measles vaccination, coverage is a key component of the cohort analyses that estimate the build-up of susceptible children after vaccination campaigns and that identify when follow-up campaigns are needed [50]. Inflated coverage data contribute to delays in implementing follow-up campaigns, which may lead to measles outbreaks [51]. Finally, coverage data can also be used in field evaluation of vaccine effectiveness [52].

Despite these important public health applications of vaccination coverage data, many challenges face the collection of accurate coverage data [5],[6] that must be addressed to guide program implementation and to ensure that funding decisions are based on real performance. For example, although survey data are often preferred to administrative reports for the reasons that we outlined at the start of this article, surveys are also subject to errors (Table 2). Groups that may be omitted from sampling frames—for example, those living in conflict-affected or otherwise inaccessible areas or in rapidly growing urban areas [53], and recent migrants—may have lower vaccination coverage [1]. Depending on the size of these population groups and the purposes of the evaluation, substantial extra effort may be needed to update the sampling frame or to conduct special surveys among these groups.

Among potential sources of non-sampling error, one of the most critical challenges is the availability of HBRs, which has been low in surveys of some of the countries contributing most to global estimates of vaccination coverage [54]. In India, WHO and UNICEF coverage estimates since 2009 have been based on a 2008 coverage evaluation survey in which vaccine cards were seen for only 52% of children [55]; in Nigeria, HBRs were seen in 40% of children in the 2009 national immunization coverage survey and in only 26% in the 2007 DHS [56]. Obtaining records of vaccines administered after infancy, including booster doses and vaccines targeted to older age groups such as tetanus toxoid, which is administered to pregnant women, is even more difficult. Records of administration of vaccines containing tetanus toxoid may be available for the most recent pregnancy but not for previous pregnancies, mass campaigns, or early childhood. Serological surveys show that vaccination coverage surveys tend to underestimate the prevalence of protection against neonatal tetanus [57]. Thus, unless primary recording of vaccination data on HBRs and clinic records is improved, investment in other strategies to improve survey methods or administrative estimates will have limited effect.

When choosing a survey methodology, decision makers should consider both the specific information needed and the speed with which the information is required. The timing and frequency of DHS and MICS surveys are not decided by immunization program managers, and thus data are used by these health professionals as and when they become available (which is several months after survey completion even for the preliminary results). In the years between DHS and MICS surveys, program managers may need additional information on vaccination coverage to improve planning and may prefer the quicker and less expensive EPI surveys; we propose ways to improve these below. Sometimes, vaccination coverage estimates are needed very rapidly (e.g., during campaigns, while the vaccination team is still in the area), and a purposive sampling method focused on areas most likely to have low coverage has been used [9],[58]. Recent experience with rapid monitoring of polio campaigns using purposive sampling, however, has not been positive, and LQAS surveys are now promoted for polio campaign monitoring [31].

Although biomarkers are under consideration for validating coverage, they have several limitations for this purpose and are not currently included routinely in surveys measuring childhood vaccination coverage. For most vaccines, the presence of antibody following vaccination cannot be distinguished from that following “natural” infection. Exceptions are the presence of tetanus antibody (infection does not generate lasting immunity) and the presence of antibodies to antigens included in subunit vaccines such as hepatitis B but not to other antigens found in the whole organism (an indication of vaccine-induced immunity) [59]. Even for these vaccines, detection of antibodies does not indicate reliably how many doses have been received [60]. Furthermore, absence of antibody means either that the child is unvaccinated or that vaccines have lost their potency. Biomarkers are therefore potentially useful to estimate population-level protection [61] but not necessarily to validate coverage measurements or vaccination program performance. The development of antibody assays on oral fluid samples for tetanus [60] and measles [62] may make surveys with repeated sample collection more acceptable and may allow evaluation of vaccination campaigns [63].

Recommendations and Conclusions

To reduce bias in coverage measurement by any method, we recommend that primary recording of vaccination data be improved. In the long term, we recommend further investment in the development, evaluation, and roll-out of effective systems for digital recording and data transmission. In the short-to-medium term in low- and middle-income countries, paper-based recording must be improved. Record design will need to evolve rapidly to accommodate the introduction of new and underutilized vaccines and to allow recording of doses administered through campaigns and across the life course. We recommend that research be conducted to improve the design of primary records to reduce recording errors and facilitate compilation of data. We also recommend that health workers educate mothers so that they value HBRs, understand the information therein, and retain them safely.

Whenever surveys are conducted, all efforts to ensure high-quality data should be made (Table 2). In particular, probability samples should be used, the sample size and number of clusters should be selected appropriately for the survey objectives, efforts should be made to encourage mothers to have HBRs ready at the time of the survey, households should be revisited if necessary to interview a suitable respondent and see the HBR, health facilities should be visited to look for records of children whose HBRs were unavailable, and strict quality control measures should be implemented. Analyses should incorporate internal consistency checks as part of quality control, including assessment of the validity of the verbal history of vaccination. These checks include comparison of prevalence of a Bacille Calmette Guerin vaccine against tuberculosis (BCG) scar among children with documented versus verbal history of BCG [32], and comparison of coverage among those who reported receiving a vaccination card but did not present it and those who presented a card [45]. In multi-indicator surveys, vaccination coverage could be cross-tabulated against coverage of other interventions for persons with and without a HBR; if the verbal history is reliable, the same associations should be found in both groups. Large-scale survey programs should also be evaluated periodically to sustain attention to quality control [64].

Technical expertise for collecting high-quality data and interpreting and using results needs to be further developed at national and sub-national levels. Program managers need to use coverage data with other program indicators to improve program planning and management. Identification of low-coverage areas should trigger action to reach underserved children, who are often those at highest risk of dying should they acquire a vaccine-preventable infection [65]. As a measure of population protection, coverage is currently limited by assumptions about vaccine effectiveness and thus is helpful but not sufficient. Additional information from vaccine management assessments, surveillance, outbreak investigations, and, where available, special studies such as case control studies of vaccine effectiveness should be reviewed together with coverage data to obtain a fuller picture of program success. Sero-surveys and vaccination coverage surveys are likely to complement each other for the foreseeable future, and we recommend that further research into the use of sero-surveys and the development of new biomarkers be undertaken.

Sources of uncertainty in surveys must be considered before drawing strong conclusions about their results. It is common to focus on the point estimate of coverage from the survey, but it is also important to consider uncertainty due to sampling design, which is usually expressed using a confidence interval, and potential biases, which should be assessed by reviewing information about the survey protocol and its implementation. The recent inclusion of a “grade of confidence” in national immunization coverage estimates produced by WHO-UNICEF is an important first step towards improving the usefulness of these estimates, and we recommend that grading of the quality of surveys should also be done. In particular, when LQAS surveys are done, we recommend that their results be interpreted with due recognition of the gray area [66].

Finally, as we mentioned at the start of this review, there is currently tension between financing systems, which reward high coverage, and efforts to improve the quality of coverage measurement. We believe and strongly recommend that it is time to reward actions that improve the quality of data, particularly those discussed in this review, rather than rewarding a country's apparent coverage achievements.

Key Points

  • Vaccination coverage is an important indicator of public health if measured accurately; at present, well-designed and executed surveys provide more accurate and comparable results than administrative reports, which are subject to incomplete and inaccurate reporting of the numerator and inaccurate estimation of the denominator.
  • To reduce bias in coverage measurement based on surveys and on administrative reports, primary recording of vaccination data on home-based records and clinic records must be improved. In the long term, this will involve digital recording and data transmission. In the short-to-medium term in low-income countries, paper-based recording must be improved.
  • Whenever surveys are done, to minimize selection bias and information bias, the sample size should be selected according to program needs, probability sampling should be used, and strict quality control measures should be implemented for data collection and analysis.
  • The potential magnitude of bias in surveys must be assessed before results are interpreted, quality assessment criteria should be developed and endorsed by partners, and partners should consider uncertainty in coverage estimates before basing decisions such as those involving performance-based financing on coverage.
  • To improve program performance, national immunization programs and their partners should take action to improve the collection, interpretation, and use of vaccination coverage data together with data on other indicators.

Acknowledgments

We thank Fred Arnold, Tony Burton, Thomas Cherian, Thom Eisele, and Attila Hancioglu for comments on earlier drafts of this work.

Author Contributions

Conducted literature review: FTC. Analyzed the data: FTC DAR. Wrote the first draft of the manuscript: FTC DAR. Contributed to the writing of the manuscript: FTC HSI DAR. ICMJE criteria for authorship read and met: FTC HSI DAR. Agree with manuscript results and conclusions: FTC HSI DAR.

References

  1. 1. Hadler SC, Dietz V, Okwo-Bele JM, Cutts FT (2008) Vaccination programs in developing countries. In: Plotkin SA, Orenstein WA, editors. Vaccines, 5th edition. Philadelphia: Saunders.
  2. 2. Brenzel L, Measham A, Naimoli J, Batson A, Bredenkamp C, et al.. (2009) Taking stock: World Bank experience with results-based financing (RBF) for health. Washington: The World Bank. Available: http://www.rbfhealth.org/rbfhealth/libra​ry/doc/taking-stock-world-bank-experienc​e-results-based-financing-rbf-health. Accessed 29 August 2012.
  3. 3. Millenium Challenge Corporation (2011) Report on the criteria and methodology for determining the eligibility of candidate countries for Millennium Challenge Account assistance in fiscal year 2012. Available: https://www.mcc.gov/documents/reports/re​port-2011001066201-fy12-selection-criter​ia.pdf. Accessed 29 August 2012.
  4. 4. GAVI Alliance (2012) Country eligibility policy. Available: http://www.gavialliance.org/about/govern​ance/programme-policies/country-eligibil​ity/. Accessed 29 August 2012.
  5. 5. Lim SS, Stein DB, Charrow A, Murray CJL (2008) Tracking progress towards universal childhood immunisation and the impact of global initiatives: a systematic analysis of three-dose diphtheria, tetanus, and pertussis immunisation coverage. Lancet 372: 2031–2046. doi: 10.1016/s0140-6736(08)61869-3
  6. 6. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, et al. (2003) Validity of reported vaccination coverage in 45 countries. Lancet 362: 1022–1027. doi: 10.1016/s0140-6736(03)14411-x
  7. 7. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A (2009) Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Health 14: 2–10. doi: 10.1111/j.1365-3156.2008.02181.x
  8. 8. Burton T, Neil M, Okwo-Bele JM, Salama P, Wardlaw T (2009) Measurement of immunisation coverage. Lancet 373: 210–211. doi: 10.1016/s0140-6736(08)61895-4
  9. 9. Dietz V, Venczel L, Izurieta H, Stroh G, Zell ER, et al. (2004) Assessing and monitoring vaccination coverage levels: lessons from the Americas. Rev Panam Salud Publica 16: 432–442. doi: 10.1590/s1020-49892004001200013
  10. 10. Hull B, Dey A, Mahajan D, Menzies R, McIntyre PB (2011) Immunisation coverage annual report, 2009. Commun Dis Intell Q Rep 35: 132–148.
  11. 11. Ronveaux O, Arrieta F, Curto S, Laurani H, Danovaro-Holliday M (2009) Assessment of the quality of immunization data produced by the national individual registration system in Uruguay, 2006. Rev Panam Salud Publica 26: 153–160. doi: 10.1590/s1020-49892009000800008
  12. 12. Luhm KR, Cardoso MRA, Waldman EA (2011) Vaccination coverage among children under two years of age based on electronic immunization registry in Southern Brazil. Rev Saude Publica 45: 90–98.
  13. 13. Crowcroft NS (2009) Action on immunisation: no data, no action. Arch Dis Child 94: 829–830. doi: 10.1136/adc.2008.138776
  14. 14. PATH (2012) Rethinking the vaccine supply chain. Available: http://www.path.org/projects/project-opt​imize. Accessed 29 August 2012.
  15. 15. Burton A, Monasch R, Lautenbach B, Gacic-Dobo M, Neill M, et al. (2009) WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bull World Health Organ 87: 535–541. doi: 10.2471/blt.08.053819
  16. 16. MEASURE DHS Demographic and Health Surveys. Available: http://www.measuredhs.com/. Accessed 1 June 2012.
  17. 17. UNICEF. Statistics and Monitoring: Multiple Indicator Cluster Survey. Available: http://www.unicef.org/statistics/index_2​4302.html. Accessed 29 June 2012.
  18. 18. Hancioglu A, Arnold F (2013) Measuring coverage in MNCH: Tracking progress in health for women and children using DHS and MICS household surveys. PLoS Med 10: e1001391 doi:10.1371/journal.pmed.1001391.
  19. 19. Henderson RH, Sundaresan T (1982) Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bull World Health Organ 60: 253–260.
  20. 20. World Health Organization (1991) Facilitator guide for the EPI coverage survey. Available: http://www.who.int/immunization_monitori​ng/routine/Facilitator_guide_EPI_coverag​e_survey.pdf. Accessed 3 January 2013.
  21. 21. World Health Organization (2005) Immunization coverage cluster survey—reference manual. Available: http://whqlibdoc.who.int/hq/2005/WHO_IVB​_04.23.pdf. Accessed 29 October 2012.
  22. 22. Grais RF, Rose AMC, Guthmann JP (2007) Don't spin the pen: two alternative methods for second-stage sampling in urban cluster surveys. Emerg Themes Epidemiol 4: 8. doi: 10.1186/1742-7622-4-8
  23. 23. Luman ET, Worku A, Berhane Y, Martin R, Cairns L (2007) Comparison of two survey methodologies to assess vaccination coverage. Int J Epidemiol 36: 633–641. doi: 10.1093/ije/dym025
  24. 24. Milligan P, Alpha N, Bennett S (2004) Comparison of two cluster sampling methods for health surveys in developing countries. Int J Epidemiol 33: 469–476. doi: 10.1093/ije/dyh096
  25. 25. Myatt M, Feleke T, Sadler K, Collins S (2005) A field trial of a survey method for estimating the coverage of selective feeding programmes. Bull World Health Organ 83: 20–26.
  26. 26. Turner AG, Magnani RJ, Shuaib M (1996) A not quite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) cluster survey design. Int J Epidemiol 25: 198–203. doi: 10.1093/ije/25.1.198
  27. 27. World Health Organization (2012) Sample design and procedures for hepatitis B immunization surveys: a companion to the WHO cluster survey reference manual. Available: http://whqlibdoc.who.int/hq/2011/WHO_IVB​_11.12_eng.pdf. Accessed 29 August 2012.
  28. 28. Cotter B, Bremer V, Stroh G, Msambichaka K, Mabuzane E, et al. (2003) Assessment of neonatal tetanus elimination in an African setting by lot quality assurance cluster sampling (LQA-CS). Epidemiol Infect 130: 221–226. doi: 10.1017/s0950268802008233
  29. 29. Pezzoli L, Pineda S, Halkyer P, Crespo G, Andrews N, et al. (2009) Cluster-sample surveys and lot quality assurance sampling to evaluate yellow fever immunisation coverage following a national campaign, Bolivia, 2007. Trop Med Int Health 14: 355–361. doi: 10.1111/j.1365-3156.2009.02231.x
  30. 30. Kim SH, Pezzoli L, Yacouba H, Coulibaly T, Djingarey MH, et al. (2012) Whom and where are we not vaccinating? Coverage after the introduction of a new conjugate vaccine against group A meningococcus in Niger in 2010. PLoS ONE 7: e29116 doi:10.1371/journal.pone.0029116.
  31. 31. World Health Organization (2012) Assessing vaccination coverage levels using clustered Lot Quality Assurance Sampling: field manual—version edited for the Global Polio Eradication Initiative (GPEI). Available: http://www.polioeradication.org/Portals/​0/Document/Research/OPVDelivery/LQAS.pdf. Accessed 29 August 2012.
  32. 32. Jahn A, Floyd S, Mwinuka V, Mwafilaso J, Mwagomba D, et al. (2008) Ascertainment of childhood vaccination histories in northern Malawi. Trop Med Int Health 13: 129–138. doi: 10.1111/j.1365-3156.2007.01982.x
  33. 33. Cutts FT, Smith PG, Colombo S, Mann G, Ascherio A, et al. (1990) Field evaluation of measles vaccine efficacy in Mozambique. Am J Epidemiol 131: 349–355.
  34. 34. World Health Organization (2012) World immunization week 2012. Available: http://www.who.int/immunization/newsroom​/events/immunization_week/en/index.html. Accessed 15 May 2012.
  35. 35. Gareaballah ET, Loevinsohn BP (1989) The accuracy of mother's reports about their children's vaccination status. Bull World Health Organ 67: 669–674.
  36. 36. Langsten R, Hill K (1998) The accuracy of mothers' reports of child vaccination: evidence from rural Egypt. Soc Sci Med 46: 1205–1212. doi: 10.1016/s0277-9536(97)10049-1
  37. 37. Valadez JJ, Weld LH (1992) Maternal recall error of child vaccination status in a developing nation. Am J Public Health 82: 120–122. doi: 10.2105/ajph.82.1.120
  38. 38. Clark A, Sanderson C (2009) Timing of children's vaccinations in 45 low-income and middle-income countries: an analysis of survey data. Lancet 373: 1543–1549. doi: 10.1016/s0140-6736(09)60317-2
  39. 39. Dayan GH, Shaw KM, Baughman AL, Orellana LC, Forlenza R, et al. (2006) Assessment of delay in age-appropriate vaccination using survival analysis. Am J Epidemiol 163: 561–570. doi: 10.1093/aje/kwj074
  40. 40. Luman ET, Barker LE, Shaw KM, McCauley MM, Buehler JW, et al. (2005) Timeliness of childhood vaccinations in the United States. JAMA 293: 1204–1211. doi: 10.1001/jama.293.10.1204
  41. 41. Babirye JN, Engebretsen IMS, Makumbi F, Fadnes LT, Wamani H, et al. (2012) Timeliness of childhood vaccinations in Kampala Uganda: a community-based cross-sectional study. PLoS ONE 7: e35432 doi:10.1371/journal.pone.0035432.
  42. 42. Eisele T, Rhoda DA, Cutts FT, Keating J, Ren R, et al. (2013) Measuring coverage in MNCH: Total survey error and the interpretation of intervention coverage estimates from household surveys. PLoS Med 10: e1001386 doi:10.1371/journal.pmed.1001386.
  43. 43. Bennett S, Radalowicz A, Vella V, Tomkins A (1994) A computer simulation of household sampling schemes for health surveys in developing countries. Int J Epidemiol 23: 1282–1291. doi: 10.1093/ije/23.6.1282
  44. 44. Lemeshow S, Tserkovnyi AG, Tulloch JL, Dowd JE, Lwanga SK, et al. (1985) A computer simulation of the EPI survey strategy. Int J Epidemiol 14: 473–481. doi: 10.1093/ije/14.3.473
  45. 45. Brown J, Monasch R, Bicego G, Burton A, Boerma JT (2002) An assessment of the quality of national child immunization coverage estimates in population-based surveys. Chapel Hill (North Carolina): MEASURE Evaluation, Carolina Population Center, University of North Carolina at Chapel Hill. Available: http://www.cpc.unc.edu/measure/publicati​ons/wp-02-53/at_download/document. Accessed 29 August 2012.
  46. 46. Little RJA, Rubin DB (2002) Statistical analysis with missing data. Hoboken (New Jersey): John Wiley & Sons.
  47. 47. Schafer JL (1997) Analysis of incomplete multivariate data. Boca Raton (Florida): Chapman & Hall/CRC.
  48. 48. Rutstein SO, Rojas G (2006) Guide to DHS statistics. Calverton (Maryland): Demographic and Health Surveys, ORC Macro.
  49. 49. Bosch-Capblanch X, Banerjee K, Burton A (2012) Unvaccinated children in years of increasing coverage: how many and who are they? Evidence from 96 low- and middle-income countries. Trop Med Int Health 17: 697–710. doi: 10.1111/j.1365-3156.2012.02989.x
  50. 50. de Quadros CA, Olive JM, Hersh BS, Strassburg MA, Henderson DA, et al. (1996) Measles elimination in the Americas. Evolving strategies. JAMA 275: 224–229. doi: 10.1001/jama.1996.03530270064033
  51. 51. Simons E, Ferrari M, Fricks J, Wannemuehler K, Anand A, et al. (2012) Assessment of the 2010 global measles mortality reduction goal: results from a model of surveillance data. Lancet 379: 2173–2178. doi: 10.1016/s0140-6736(12)60522-4
  52. 52. Orenstein WA, Bernier RH, Dondero TJ, Hinman AR, Marks JS, et al. (1985) Field evaluation of vaccine efficacy. Bull World Health Organ 63: 1055–1068 Available: http://www.ncbi.nlm.nih.gov/pmc/articles​/PMC2536484/pdf/bullwho00089-0084.pdf. Accessed 29 August 2012.
  53. 53. Bharti N, Tatem AJ, Ferrari MJ, Grais RF, Djibo A, et al. (2011) Explaining seasonal fluctuations of measles in Niger using nighttime lights imagery. Science 334: 1424–1427. doi: 10.1126/science.1210554
  54. 54. Brown DW (2012) Child immunization cards: essential yet underutilized in national immunization programmes. Open Vaccine J 5: 1–7. doi: 10.2174/1875035401205010001
  55. 55. World Health Organization (2011) India: WHO and UNICEF estimates of immunization coverage: 2011 revision. Available: http://www.who.int/immunization_monitori​ng/data/ind.pdf. Accessed 29 June 2012.
  56. 56. World Health Organization (2011) Nigeria: WHO and UNICEF estimates of immunization coverage: 2011 revision. Available: http://www.who.int/immunization_monitori​ng/data/nga.pdf. Accessed 29 August 2012.
  57. 57. World Health Organization (2006) Tetanus vaccine WHO position paper. Wkly Epidemiol Rec 85: 197–208.
  58. 58. Izurieta H, Venczel L, Dietz V, Tambini G, Barrezueta O, et al. (2003) Monitoring measles eradication in the region of the Americas: critical activities and tools. J Infect Dis 187: S133–S139. doi: 10.1086/368028
  59. 59. Cutts F, Hall A (2004) Vaccines for neonatal viral infections: hepatitis B vaccine. Expert Rev Vaccines 3: 349–352 Accessed 29 August 2012. doi: 10.1586/14760584.3.4.349
  60. 60. Tapia MD, Pasetti MF, Cuberos L, Sow SO, Doumbia MN, et al. (2006) Measurement of tetanus antitoxin in oral fluid: a tool to conduct serosurveys. Pediatr Infect Dis J 25: 819–825. doi: 10.1097/01.inf.0000232629.72160.bb
  61. 61. Fortuin M, Maine N, Mendy M, Hall A, George M, et al. (1995) Measles, polio and tetanus toxoid antibody levels in Gambian children aged 3 to 4 years following routine vaccination. Trans R Soc Trop Med Hyg 89: 326–329. doi: 10.1016/0035-9203(95)90564-2
  62. 62. Nigatu W, Nokes D, Enquselassie F, Brown D, Cohen B, et al. (1999) Detection of measles specific IgG in oral fluid using an FITC/anti-FITC IgG capture enzyme linked immunosorbent assay (GACELISA). J Virol Methods 83: 135–144. doi: 10.1016/s0166-0934(99)00116-0
  63. 63. Nigatu W, Samuel D, Cohen B, Cumberland P, Lemma E, et al. (2008) Evaluation of a measles vaccine campaign in Ethiopia using oral-fluid antibody surveys. Vaccine 26: 4769–4774. doi: 10.1016/j.vaccine.2008.07.005
  64. 64. Ahmed S, Ali D, Bisharat L, Hill A, LaFond A, et al.. (2009) Evaluation of UNICEF Multiple Indicator Cluster Surveys round 3 (MICS3) final report. Boston (Massachusetts): John Snow. Available: http://www.unicef.org/evaldatabase/files​/MICS.pdf. Accessed 29 August 2012.
  65. 65. Rheingans R, Atherly D, Anderson J (2012) Distributional impact of rotavirus vaccination in 25 GAVI countries: estimating disparities in benefits and cost-effectiveness. Vaccine 30 (Suppl 1) A15– A23 Available: http://ac.els-cdn.com/S0264410X12000333/​1-s2.0-S0264410X12000333-main.pdf?_tid=4​1e1f1f9a28236cbe2409273fa8fecc4&acdnat=1​340737914_ce1e78df52b569cf4a045350c37388​5c. Accessed 29 August 2012.
  66. 66. Rhoda DA, Fernandez SA, Fitch DJ, Lemeshow S (2010) LQAS: user beware. Int J Epidemiol 39: 60–68. doi: 10.1093/ije/dyn366
  67. 67. World Health Organization (2012) Table 2: summary of WHO position papers—recommended routine immunizations for children. Available: http://www.who.int/immunization/policy/I​mmunization_routine_table2.pdf. Accessed 23 July 2012.