The Millennium Village Project: a correction

Paul Pronyk, lead author of the Millennium Village paper, releases the following statement about the paper, Pronyk PM, Muniz M, Nemser B, et al, for the Millennium Villages Study Group. The effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment. Lancet 2012; published online May 8. DOI:10.1016/S0140-6736(12)60207-4.

“In our paper on the effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa (published online May 8), we made some erroneous statements and assumptions.

First, the statement in the Findings section of the Summary that “The average annual rate of reduction of mortality in children younger than 5 years of age was three-times faster in Millennium Village sites than in the most recent 10-year national rural trends (7·8% vs 2·6%)” is unwarranted and misleading. The paper compares the 2006 Millennium Village baseline with the 3-year results during 2006–09, a decline of 22%, to calculate an annual average decline in the under-5 mortality rate. This rate is compared, erroneously, with the under-5 mortality rate for the years 2001–10 with Demographic and Health Surveys (DHS), rather than DHS surveys for years 2006 and 2009, or thereabouts as available, which would be the correct comparison. Presentation of the earlier base period of 2001 for the DHS data underestimates national rural rates of mortality decline, thereby putting the Millennium Village mortality reduction in an excessively positive light. We withdraw the claim (which is repeated in the Results section).

Second, the decline of 22% represents an annual decline of 7·8% only if it is assumed that the decline took place over the 3 years of the project. The study team made this assumption to balance several considerations, including a likely underestimation of the baseline mortality rate due to recall error. Nonetheless, on further consideration, a more appropriate and conservative assumption is that the 22% decline occurred over 4 years rather than 3 years, implying an annualised rate of decline of 5·9%.

The paper presents credible evidence about the pace of decline in the Millennium Villages relative to matched comparison sites, as per the study protocol, not relative to national averages. We will update our assessment of the Millennium Village performance vis-à-vis national trends using the additional information on more recent years of the Millennium Villages project and the DHS. At this point all we can note is that rapid progress is being made nationally as well as in the Millennium Villages, and most likely for similar reasons, including the scaling up of effective low-cost interventions such as the mass distribution of antimalarial bednets.

The endpoint under-5 mortality goal of the Millennium Villages project is an under-5 mortality rate of less than or equal to 40 per 1000 by the end of 2015, to achieve Millennium Development Goal 4. The project will strive to report highly accurate and timely information about the trajectory of the Millennium Villages vis-à-vis this endpoint, and vis-à-vis national trends, using mortality recall data (as in DHS and The Lancet paper) in conjunction with real-time vital-events data, to improve the accuracy of the measurement of the under-5 mortality rate. The project will invite an independent panel of experts, including critics of the project, to participate in scrutinising the vital events and survey data and in assessing their validity.”

The Editors of The Lancet respond in a Letter: “The Millennium Villages project is an ambitious decade-long effort to find solutions to predicaments facing millions of people living in extreme poverty. Conceived as a means to harness multiple interventions—across health, education, and agriculture, among other sectors—in the quest to improve the lives and livelihoods of these most disadvantaged populations, the project has attracted considerable attention and controversy. That controversy has continued with the recent publication of an interim analysis of Millennium Villages by Paul Pronyk and colleagues.

Almost immediately after the paper was published online on May 8, criticism appeared suggesting that the research team’s study design and analysis were weak, and that they were less than transparent with their data and methods. The project, will address these criticisms in the fora where they were initially made. Several of these criticisms are misinformed. But in The Lancet, the authors now correct one error that did appear in the published article. They are withdrawing the comparison they made between reductions in under-5 mortality in Millennium Villages and national rural trends in child mortality.

The Millennium Villages project team has quickly and commendably corrected the record after understanding the validity of the challenge it received. But the withdrawal of this element of the paper does not detract from the larger result—namely, that after 3 years Millennium Villages saw falls in poverty, food insecurity, stunting, and malaria parasitaemia, together with increases in access to safe water and sanitation. In addition, this correction does not change the finding that under-5mortality rates were significantly lower in Millennium Village sites than their baseline rates and those of matched comparison sites.

As was made clear in the accompanying Comment by Grace Malenga and Malcolm Molyneux, these interim findings should be judged with appropriate caution. The comparison between Millennium and matched villages was not randomised, and so no causal attribution of measured changes can be conclusively ascribed to the interventions. That said, the results presented, in the words of one of the original reviewers of the paper, are “encouraging evidence—a big intervention rapidly leading to measurable results.” And, in fairness to those attempting to evaluate the effects of complex health programmes, non-randomised comparisons do have the power to illuminate—one example being the retrospective non-randomised assessment of UNICEF’s Accelerated Child Survival and Development programme in west Africa.

The next phase of the Millennium Villages project will involve the monitoring of actual vital events (instead of relying on recall). To ensure that all future data from the project are fully and fairly evaluated,  Prof Jeffrey Sachs, the Principal Investigator of the Millennium Villages project, is establishing new internal and external oversight procedures, including the creation of an International Scientific Expert Advisory Group, chaired by Professor Robert Black, Chairman of the Department of International Health, Johns Hopkins Bloomberg School of Public Health, which will report to the Principal Investigator and also communicate its findings to The Lancet. The goal is to provide a further independent means of verifying the quality of the project’s design and analysis. It is important that this work, which is of considerable significance for understanding how countries scale up multiple complex interventions across sectors, receives proper scientific evaluation before, during, and after publication.”