The challenge
You have packed your suitcases, got up early to go to the airport to catch your flight, went through security smoothly, and now you are waiting for your flight to depart. Twenty more minutes, ten more minutes, and suddenly… Your flight is delayed. A not-so-great start to kick off your relaxing holiday. Unfortunately, it is a reality that around 18% of flights fail to depart on time due to a variety of logistical troubles and punctuality defeaters, such as bad weather, staffing shortages, and air traffic issues.
EUROCONTROL, a civil-military organization, dedicated to supporting European aviation, is continuously looking for ways to deal with these challenges. They deliver solutions across the full spectrum of air traffic management. In this case, EUROCONTROL originally wanted to predict capacity issues at the airport. After analyzing the available data, we decided to predict flight regulations in European airports, as this was considered one of the most constraining factors for capacity issues. To tackle this particular issue, EUROCONTROL reached out to Brainjar to build an artificial intelligence-powered solution.
As a passenger, you often have no idea why your flight has been delayed. So it may come as a surprise that frequently, flights are delayed or even canceled because of flight regulations. These regulations are coordinated by local authorities and the EUROCONTROL Network Manager Operations Centre (NMOC). A limited amount of flights (or even none) are allowed to land or take off for a certain period when these are issued. The local authorities and NMOC can issue these regulations due to multiple factors: bad weather, industrial actions, air traffic controller shortages, limited aerodrome capacity, and so on. These factors can cause the airport resources such as the runway, parking positions, terminal, and so on, to get too crowded, possibly exceeding the maximum capacity.
By issuing a regulation, airports reduce the risk of accidents, as air traffic controllers cannot adequately monitor the airspace when maximum capacity is exceeded. Sure, delays are bothersome, but safety always goes first. What EUROCONTROL wanted to do was predict these regulations and anticipate them. As this can be modeled as a time series problem, a machine learning solution was the perfect fit.
The assignment was clear: build a model that can predict regulations for the top 100 biggest European airports. However, there were several big challenges to overcome. Due to the many different variables, we had to select the most relevant data sources and transform and feed them to the AI model. While obtaining data might seem straightforward, it usually is not, as we have to start from unstructured data.
So, we started the project by developing a model for one airport with a clear proof-of-concept: predict regulations that weather conditions may cause. Flight regulations due to weather are most predictable: for example, when the weather is terrible, chances are that visibility is limited. First, we used historical weather data. Then, we added historical data regarding the number of flights. As we continued, we gradually added steps: we changed the historical data for prediction data. Think of the number of flights, notifications of work (think of instances where there is a crane in the vicinity of the runway), and ultimately, we integrated the four most prominent causes for regulations successfully:
Of course, there are other causes for flight regulations, such as incidents or accidents, but these are either unpredictable or do not happen often enough to incorporate into the model. So, when these indicators could predict the regulations for our first airport, we gradually extended the model to the top 100 European airports.
One challenge we had to overcome was the imbalance across airports. Some airports have numerous regulations — they might even be infamous for their delays — and others hardly have any. The golden rule for artificial intelligence is that you get more accurate predictions when you add more data. So, it is logical that the model works better for those airports that often have regulations (as there is a lot of data there). To get a little more technical, we trained a convolutional neural network that predicts the probability of a regulation for the next 24 hours at an airport. To tackle the imbalance issues, we used custom loss functions and implemented our own oversampling algorithm to create additional synthetic data. We won’t go into many details here, but if you’re interested in how this works, let us know!
Even though airports today are up-and-running again, we did encounter a significant impact of Covid-19 on our models. As we started this project in 2020, we used data from ‘18 - ‘19 for the first iteration and eventually developed an accurate model. Of course, the moment we wanted to put the model into production, something happened no one could predict — the Covid-19 pandemic. Flights were canceled, and the entirety of air traffic was changed drastically.
While the training data was made up of 30.000 flights per day, during the pandemic, there were less than a mere 10.000 flights per day. When data changes this drastically, it significantly impacts AI models. Therefore, we opted for mixed data training for a new and improved model. We still used the ‘18 - ‘19 data (as we suspected that air traffic would be back up one day) and decided to use data from the pandemic time frame. However, we used only the data after June 2021, as traffic started to go up again and has kept increasing ever since.
As machine learning algorithms — especially neural networks — are known to be black boxes, and we wanted some insights into the predictions, we used something else to explain the predictions. Using SHAP values (a concept originally from cooperative game theory), we obtained more explainability of our model, which led to more trust. In the end, our model’s accuracy was around 90%, and the F1 score was 78%. Simply put, the F1-score is a weighted average of precision (so, of all predictions, how many are correct?) and recall (and of all regulations, how many are predicted?).
After working on this solution for a long time, it was time to put it to work. Of course, our model could predict the regulations, but we also wanted people to have a clear overview of when and where a regulation would be issued. So, we developed a Power BI dashboard that users can easily use to see the visualizations of the predictions.
This solution fits right into our human-in-the-loop philosophy. Just because we can accurately predict regulations now does not mean that the problem has magically disappeared. The prediction platform, however, is a tool for people to see faster when such regulations will occur and therefore anticipate where the potential bottlenecks and problems in the network will emerge or even take preventive measures to avoid them.
Of course, now we keep looking for ways to improve the model. Due to the uncertainty of the forecast data, we will never have a 100% accurate model (for instance, weather forecasts are not always accurate). This has a significant impact. So, we try to improve in ways we can make a difference, for instance, by providing an even better transformation of work notifications. The beauty of artificial intelligence is that we can evaluate the model and retrain it as often as needed.
Moreover, we are looking into predicting how long flights will be delayed as well. Because what’s worse than a delay? Not knowing how long it will be. This way, hopefully, your next flight delay to your relaxing holiday is negligible.
"The EUROCONTROL Network Manager is using artificial intelligence and machine learning to better predict capacity reductions that are likely to affect Europe’s largest airports and in consequence the air traffic flow throughout the entire Network. State-of-the-art applications such as this enable us to check and coordinate immediately with all actors to agree on the most appropriate measures that will minimise delay.”
Want to know more?
