On 20 October, Facebook Artificial Intelligence (Facebook AI) began publishing forecasts that predict the spread of COVID-19 in over 3,000 US counties. These forecasts are leveraging Facebook’s Data for Good tools, including Symptom Survey and Movement Range Maps. They are available on Humanitarian Data Exchange, and more information about the effort is published on their Data for Good site.

The core of the algorithm is a neural network that tries to predict the evolution of COVID-19 from past data. The neural network must not only learn from the evolution of epidemics in the past but also try to unveil how their evolution depended on mobility patterns, weather and human behaviour. Right now, modelling human responses is close to impossible with a mechanistic approach, that is, knowing in advance how it is going to react. Given this obstacle, artificial intelligence is considered to be one of the best options in empirical modelling given its ability to infer changes in future behaviour from past behaviour. Full details of the algorithm and how it is trained have been published in open access.

The results of the AI forecast in US counties have shown strong performance compared to state-of-the-art models in their three-week forecasts, but a detailed calibration is pending. The Computational Biology and Complex Systems Group (BIOCOM-SC) linked to the Department of Physics of the Universitat Politècnica de Catalunya · BarcelonaTech (UPC) will collaborate with Facebook AI in training the model to forecast the evolution of the disease in Spain. Enric Álvarez and PhD student David Conesa will be in charge of this training, alongside Facebook research scientist Maximilian Nickel. Proper training is crucial since, as  Álvarez says, “the fact that the AI gives good results in the US does not mean that it will immediately give good results in Spain or Italy. We have been testing the AI for a couple of months now and there is still a lot of work to do, especially regarding the introduction of proper patterns of mobility to train the AI. The mobility patterns are very different here than in the US. The relevance of mobility indicators in the quality of forecasting in southern Europe is one of our key open questions.” David Conesa says, “We are working now on which mobility index or indexes can be better for the AI. It is not easy since different indicators provide different information, so trying to combine them properly is our present work.”

Still, it is as important to calibrate the AI’s forecasts and quantify its reliability as it is to train it. Sergio Alonso of BIOCOM-SC and Martí Català of the Germans Tries i Pujol Research Institute will be in charge of performing this calibration for European data. “The proper way to assess the behaviour of the AI is to calibrate carefully the error intervals of the predictions,” says Alonso. “One of the challenges of dealing with AI is to know how to establish error bars in its predictions, in other words, how accurate you want the AI to be and what predictions are classified as failures. If you are too lax with the accuracy then the AI will most of the time get it right but it might not provide any useful information, and if you are too demanding the AI may fail too often.”

In this sense, the present evolution of the epidemic in Spain will be a key testing ground for the AI. As Martí Català, who is in charge of monitoring the calibration of the forecast in Catalonia, indicates, “the present heterogeneous, but strong, increase in cases in different parts of Spain is the perfect scenario for calibrating the AI and knowing how far in the future it can forecast. We will see how it performs with different training protocols. Right now we are not certain that it can provide accurate three-week assessment for this second wave, but it is certainly worth trying.”

Preliminary analysis and research by different groups have already shown that forecasts of any kind beyond one month seem to have low reliability. Facebook AI does not provide forecasts beyond one month either. However, a properly trained AI that can accurately predict the evolution of the epidemic two or three weeks in advance, whether subject or not to the different measures that are implemented, will be a major milestone in understanding the epidemic. But way before that, the forecasts of the AI can, with other models, provide complementary evidence on the general tendency of the epidemic. If the AI has been properly trained with local expertise, it can be useful to test different future scenarios.