How accurate is COVID-19 modeling?

By Sam HancockOct 5 2021

Coronavirus disease 2019 (COVID-19) modeling has shown itself to be drastically important to public health policy. With millions of deaths and hundreds of millions of cases worldwide, knowing the likely spread and risk of the disease helped governments apply lockdowns, social distancing measures, and other restrictions to minimize the spread of the disease and buy time for vaccine development.

Study: Predicting the unpredictable: how dynamic COVID-19 policies and restrictions challenge model forecasts. Image Credit: Bihlmayer Fotografie/ Shutterstock

This news article was a review of a preliminary scientific report that had not undergone peer-review at the time of publication. Since its initial publication, the scientific report has now been peer reviewed and accepted for publication in a Scientific Journal. Links to the preliminary and peer-reviewed reports are available in the Sources section at the bottom of this article. View Sources

During a global health and economic crisis, balancing the potential capacity for opening the economy against risking further transmission was essential. Models can also help select areas for vaccine trials by determining the geographic likelihood of high incidence rates. One particular agent-based model was used to identify the potential rates for the spread of COVID-19 between June and August 2020. Researchers from the Burnet Institute in Melbourne have been using this to investigate how accurate COVID-19 modeling has been.

A preprint version of the study is available on the medRxiv* server while the article undergoes peer review.

The study

The initial model predicts incidence and diagnosis rates for COVID-19 worldwide, examining 72 locations in Australia, Belgium, Brazil, France, Italy, Mexico, The Netherlands, South Africa, Spain, and the United States. It predicted cases for a six-week window during September and October 2020, two or three months from the date of predictions. This six-week window was the time in which the Janssen vaccine would begin clinical trials.

The model used was an agent-based model which simulates decision-making entities called agents that base their decisions on a series of rules. In this case, 100,000 agents were used representing individuals who would interact with each other. The model was calibrated for each of the 72 locations using several variables: population size, age structure, household size distribution, contact data through employment, household and school/community contacts, and global disease parameter estimates.

Interventions such as testing, contact tracing, quarantine and other restriction such as social distancing or mask usage could be implemented to alter the model. Five models were initially created – showing no change in restrictions, 10% easing of restrictions (and 10% increase in individual transmission), 20% easing of restrictions and increase in transmission, and a model with no infections imported from outside as well as a model with constant testing. Data was gathered from the simulations once the models were adjusted for accuracy with the current reality in each area.

The researchers examined how often the data fell within the upper and lower bounds initially specified and which conditions were statistically significant in the models' success or failure to assess the accuracy of their initial model. Unfortunately, 45/72 projections fell out of bounds and were not accurate.

Generally, the models that were closest to reality had correctly predicted future health policy or testing rates or longer projection periods. Other factors not involved in the prediction that proved to be beneficial for accuracy were a more youthful population or higher unemployment or poverty rates. A younger population could aid more in assuming transmission, as there will be fewer individuals in nursing homes and a generally more resistant population.

As higher deprivation index ratings are correlated with increased transmission and severity, higher poverty and unemployment could also be beneficial in predicting the spread of the disease. The most important factor by far was public health policy assumptions. The most erroneous predictions occurred when the modeling predicted the opposite restriction change to reality, such as loosening restriction when the model showed tightened restrictions. Even when the scientists correctly guessed the direction of the changes, the models were only accurate in 49% of cases.

Conclusion

The authors highlight rapid policy changes from multiple governments as one of the difficulties of modeling any disease, particularly in the COVID-19 pandemic. Particularly in the early days of the pandemic, as well as the initial loosening of restrictions, government advice could change regularly, making accurate predictions difficult. The researchers argue that this supports the need for more revision of modeling and projections as government policies change, as well as revealing the risks of modeling too far into the future. As well as this, the results show the importance of public health policies both in understanding the spread of disease and preventing that spread.

The unpredictable and sometimes unintuitive changes largely explain the failures of COVID-19 models and provide strong motivation for researchers to work with policymakers as much as possible and seek the advice of behavioral experts to determine the reaction to certain restrictions.

This news article was a review of a preliminary scientific report that had not undergone peer-review at the time of publication. Since its initial publication, the scientific report has now been peer reviewed and accepted for publication in a Scientific Journal. Links to the preliminary and peer-reviewed reports are available in the Sources section at the bottom of this article. View Sources