A team of researchers from various institutions in the UK has reviewed multi-model forecasts of the coronavirus disease 2019 (COVID-19) pandemic made between the end of March and mid-July 2020.
The forecasts had been collated and aggregated in an effort to predict the burden of the pandemic on the healthcare system and to indicate the status of the epidemic in the UK at the time.
The researchers evaluated the performance of the models and combined predictions from individual models into ensemble forecasts to maximize performance.
The team compared the performance of both the individual and ensemble models with that of a null model, where each forecasting target was assumed to remain at its current value indefinitely.
Sebastian Funk from the London School of Hygiene & Tropical Medicine and colleagues say that in the majority of cases, individual models performed better than the null model, while ensembles models performed as well as the best individual models.
The researchers say ensembles of multi-model forecasts could be used to inform the policy decisions made in response to the pandemic by evaluating future resource needs and the expected impact of morbidity and mortality on the population.
"As SARS-CoV-2 continues to spread in populations around the world, real-time modeling and short-term forecasting can be key tools for short-term resource planning and pandemic management," they write.
A pre-print version of the paper is available on the server medRxiv*, while the article undergoes peer review.
Rapid spread of COVID-19 led to lockdown in many countries
Since the COVID-19 outbreak began in Wuhan, China, late last year, the pandemic's unprecedented spread has left healthcare services overwhelmed in many parts of the world and resulted in an enormous burden of morbidity and mortality.
In the UK, the number of cases quickly rose during February and March, and on March 23rd, the government imposed lockdown measures in an effort to control transmission while hospitals prepared for the influx of severely ill patients.
Short-term forecasts of infectious diseases can aid situational awareness and planning for outbreak response, say Funk and colleagues. Such forecasts are increasingly being used to inform public health policy across a range of illnesses.
"Developing accurate and reliable short-term forecasts in real-time for novel infectious agents such as SARS-CoV-2 in early 2020 is particularly challenging because of uncertainty about modes of transmission, severity profiles, and other relevant parameters," writes the team.
What did the researchers do?
Now, the researchers have reviewed forecasts of the COVID-19 epidemic in the UK that were collated and combined using a system set up through a short-term forecasting subgroup of the Scientific Pandemic Influenza Group on Modelling (SPI-M) on March 23rd, 2020.
The system was set up in response to the urgent need for predictions of the epidemic's trajectory and the burden on the healthcare system.
Funk and colleagues reviewed multi-model forecasts that were generated at regular intervals between the end of March and July and assessed the quality of the predictions made at different times.
They assessed the performance of individual model forecasts using a variety of metrics and then combined the predictions made by these models into ensemble forecasts that could be compared to select the optimum procedure.
The performance of both individual and ensemble models was compared with that of a null model. The null model assumed no change in the current value of forecasting targets such as the number of new hospital admissions or daily deaths due to COVID-19.
Weights given to the different models in the median prediction of the best-performing QRA at each forecast date for England.
*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
What did they find?
The team reports that in the majority of cases, individual models consistently performed better than the null model (no change) when predictions applied to a 2-week time horizon. However, the null model outperformed the individual models when predictions applied to a 1-week time horizon.
While there was little consistency amongst the models in terms of performance against the targets considered and no evidence of one model clearly outperforming the others, all the ensemble models performed as well as or better than the best individual models.
"Ensembles of multi-model forecasts can inform the policy response to the COVID-19 pandemic by assessing future resource needs and expected population impact of morbidity and mortality," write the researchers.
Future forecasts could include medium-term projections
Funk and colleagues say that in the UK, similar methodologies to the ones presented here are now being used to generate medium-term projections – that is, predictions covering more than a 3 -week time horizon regarding what might happen if nothing changed from the current situation.
"As SARS-CoV-2 continues to affect populations around the world, short-term forecasts and longer-term projections can play a crucial part in real-time monitoring of the epidemic and its expected near-term impact in morbidity, healthcare utilization, and mortality," concludes the team.
*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
COVID-19's impact on early education: Retrospective study shows decrease in kindergarten readiness