Quickstart¶
The motivation for this page/notebook is to take the reader through all basic functionalities of the traffic library. We will cover:
- a basic introduction about
FlightandTrafficstructures; - how to produce visualisations of trajectory data;
- how to access basic sources of data;
- a simple use case to select trajectories landing at Toulouse airport;
- an introduction to declarative descriptions of data preprocessing through lazy iteration.
This page is also available as a notebook which can be downloaded and executed locally; or loaded and executed in Google Colab.
Basic introduction¶
The traffic library provides natural methods and attributes that can be applied on trajectories and collection of trajectories, all represented as pandas DataFrames.
The Flight structure¶
Flight is the core class offering representations, methods and attributes to single trajectories. A comprehensive description of the API is available here.
Sample trajectories are provided in the library: belevingsvlucht is
one of them, context is explained
here.
from traffic.data.samples import belevingsvlucht
belevingsvlucht
- aircraft: 484506 / PH-HZO (B738)
- from: 2018-05-30 15:21:38+00:00
- to: 2018-05-30 20:22:56+00:00
Among available attributes, you may want to access:
- its
callsign(the identifier of the flight displayed on ATC screens); - its transponder unique identification number (
icao24); - its
registrationnumber (tail number); - its
typecode(i.e. the model of aircraft).
(
belevingsvlucht.callsign,
belevingsvlucht.icao24,
belevingsvlucht.registration,
belevingsvlucht.typecode,
)
# ('TRA051', '484506', 'PH-HZO', 'B738')
Methods are provided to select relevant parts of the flight, e.g. based on timestamps.
The start and stop attributes refer to the timestamps of the
first and last recorded samples. Note that all timestamps are by default
set to universal time (UTC) as it is common practice in aviation.
(belevingsvlucht.start, belevingsvlucht.stop)
# (Timestamp('2018-05-30 15:21:38+0000', tz='UTC'),
# Timestamp('2018-05-30 20:22:56+0000', tz='UTC'))
first30 = belevingsvlucht.first(minutes=30)
after19 = belevingsvlucht.after("2018-05-30 19:00", strict=False)
# Notice the "strict" comparison (>) vs. "or equal" comparison (>=)
print(f"between {first30.start:%H:%M:%S} and {first30.stop:%H:%M:%S}")
print(f"between {after19.start:%H:%M:%S} and {after19.stop:%H:%M:%S}")
# between 15:21:38 and 15:51:37
# between 19:00:00 and 20:22:56
between1920 = belevingsvlucht.between(
"2018-05-30 19:00", "2018-05-30 20:00"
)
between1920
- aircraft: 484506 / PH-HZO (B738)
- from: 2018-05-30 19:00:01+00:00
- to: 2018-05-30 19:59:59+00:00
The underlying dataframe is always accessible.
between1920.data.head()
| timestamp | icao24 | latitude | longitude | groundspeed | track | vertical_rate | callsign | altitude | |
|---|---|---|---|---|---|---|---|---|---|
| 11750 | 2018-05-30 19:00:01+00:00 | 484506 | 52.839973 | 5.793947 | 290 | 52 | -1664 | TRA051 | 8233 |
| 11751 | 2018-05-30 19:00:02+00:00 | 484506 | 52.840747 | 5.795680 | 290 | 52 | -1664 | TRA051 | 8200 |
| 11752 | 2018-05-30 19:00:03+00:00 | 484506 | 52.841812 | 5.797501 | 290 | 52 | -1664 | TRA051 | 8200 |
| 11753 | 2018-05-30 19:00:04+00:00 | 484506 | 52.842609 | 5.799133 | 290 | 52 | -1599 | TRA051 | 8149 |
| 11754 | 2018-05-30 19:00:05+00:00 | 484506 | 52.843277 | 5.801010 | 289 | 52 | -1599 | TRA051 | 8125 |
The Traffic structure¶
Traffic is the core class to represent collections of trajectories, which are all flattened in the same pandas DataFrame. A comprehensive description of the API is available here.
We will demonstrate here with a sample of ADS-B data from the OpenSky Network.
The basic representation of a Traffic object is a summary view of the data: the structure tries to infer how to separate trajectories in the data structure based on customizable heuristics, and returns a number of sample points for each trajectory.
from traffic.data.samples import quickstart
quickstart
| count | ||
|---|---|---|
| icao24 | callsign | |
| 4ca84d | RYR3YM | 2859 |
| 393320 | AFR27GH | 2770 |
| 505c98 | RAM667 | 2752 |
| 3944ef | HOP87DJ | 2731 |
| 4ca574 | IBK5111 | 2706 |
| 393322 | AFR23FK | 2665 |
| 40643a | EZY57FT | 2656 |
| 394c18 | AFR140W | 2613 |
| 344692 | VLG2972 | 2599 |
| 400cd1 | EZY81GE | 2579 |
Traffic offers the ability to index and iterate on all flights contained in the structure. Traffic will use either:
- a combination of
timestamp,icao24(aircraft identifier) andcallsign(mission identifier); or - a customizable flight identifier (
flight_id);
as a way to separate and identify flights.
Indexation will be made on either of icao24, callsign (or
flight_id if available).
quickstart["AFR27GH"] # on callsign
quickstart["393320"] # on icao24
- aircraft: 393320 / F-GMZA (A321)
- from: 2017-07-16 19:30:00+00:00
- to: 2017-07-16 20:16:10+00:00
A subset of trajectories can also be selected if a list is passed an index:
quickstart[["AFR27GH", "HOP87DJ"]]
| count | ||
|---|---|---|
| icao24 | callsign | |
| 393320 | AFR27GH | 2770 |
| 3944ef | HOP87DJ | 2731 |
In many cases, flight_id are more convenient to access specific
flights yielded by iteration. We may construct custom flight_id:
from traffic.core import Traffic
quickstart_id = Traffic.from_flights(
flight.assign(flight_id=f"{flight.callsign}_{i:03}")
for i, flight in enumerate(quickstart)
)
quickstart_id
| count | |
|---|---|
| flight_id | |
| RYR3YM_343 | 2859 |
| AFR27GH_046 | 2770 |
| RAM667_373 | 2752 |
| HOP87DJ_055 | 2731 |
| IBK5111_316 | 2706 |
| AFR23FK_048 | 2665 |
| EZY57FT_172 | 2656 |
| AFR140W_064 | 2613 |
| VLG2972_036 | 2599 |
| EZY81GE_149 | 2579 |
.assign_id() method, which is implemented
exactly that way.eval() further below)quickstart.assign_id().eval()
| count | |
|---|---|
| flight_id | |
| RYR3YM_343 | 2859 |
| AFR27GH_046 | 2770 |
| RAM667_373 | 2752 |
| HOP87DJ_055 | 2731 |
| IBK5111_316 | 2706 |
| AFR23FK_048 | 2665 |
| EZY57FT_172 | 2656 |
| AFR140W_064 | 2613 |
| VLG2972_036 | 2599 |
| EZY81GE_149 | 2579 |
Saving and loading data¶
Some processing operations are computationally expensive and time consuming. Therefore, it may be relevant to store intermediate results in files for sharing and reusing purposes.
One option is to store Traffic and Flight underlying DataFrames in pickle format. Details about storage formats are presented here.
quickstart_id.to_pickle("quickstart_id.pkl")
from traffic.core import Traffic
# load from file again
quickstart_id = Traffic.from_file("quickstart_id.pkl")
Visualisation of data¶
traffic offers facilities to leverage the power of common visualisation renderers including Cartopy, a map plotting library built around Matplotlib, and Altair.
More visualisation renderers such as Leaflet are available as plugins.
Visualisation of trajectories¶
When you choose to plot trajectories on a map, you have to make a choice concerning how to represent points at the surface of a sphere (more precisely, an oblate spheroid) on a 2D plane. This transformation is called a projection.
The choice of the right projection depends on the data. The most basic
projection (sometimes wrongly referred to as no projection) is the
PlateCarree(), when you plot latitude on the y-axis and longitude on
the x-axis. The famous Mercator() projection distorts the latitude
so as lines with constant bearing appear as straight lines. Conformal
projections are also convenient when plotting smaller areas (countries)
as they preserve distances (locally).
Many countries define official projections to produce maps of their
territory. In general, they fall either in the conformal or in the
Transverse Mercator category. Lambert93() projection is defined over
France, GaussKruger() over Germany, Amersfoort() over the
Netherlands, OSGB() over the British Islands, etc.
When plotting trajectories over Western Europe, EuroPP() is a decent
choice.
from traffic.data.samples import airbus_tree
%matplotlib inline
import matplotlib.pyplot as plt
from traffic.core.projection import Amersfoort, GaussKruger, Lambert93, EuroPP
from traffic.drawing import countries
with plt.style.context("traffic"):
fig = plt.figure()
# Choose the projection type
ax0 = fig.add_subplot(221, projection=EuroPP())
ax1 = fig.add_subplot(222, projection=Lambert93())
ax2 = fig.add_subplot(223, projection=Amersfoort())
ax3 = fig.add_subplot(224, projection=GaussKruger())
for ax in [ax0, ax1, ax2, ax3]:
ax.add_feature(countries())
# Maximum extent for the map
ax.set_global()
# Remove border and set transparency for background
ax.outline_patch.set_visible(False)
ax.background_patch.set_visible(False)
# Flight.plot returns the result from Matplotlib as is
# Here we catch it to reuse the color of each trajectory
ret, *_ = quickstart["AFR27GH"].plot(ax0)
quickstart["AFR27GH"].plot(
ax1, color=ret.get_color(), linewidth=2
)
ret, *_ = belevingsvlucht.plot(ax0)
belevingsvlucht.plot(
ax2, color=ret.get_color(), linewidth=2
)
ret, *_ = airbus_tree.plot(ax0)
airbus_tree.plot(
ax3, color=ret.get_color(), linewidth=2
)
# We reduce here the extent of the EuroPP() map
# between 8°W and 18°E, and 40°N and 60°N
ax0.set_extent((-8, 18, 40, 60))
params = dict(fontname="Ubuntu", fontsize=18, pad=12)
ax0.set_title("EuroPP()", **params)
ax1.set_title("Lambert93()", **params)
ax2.set_title("Amersfoort()", **params)
ax3.set_title("GaussKruger()", **params)
fig.tight_layout()
Altair API is not very mature yet with geographical data, but basic visualisations are possible.
# Mercator projection is the default one with Altair
quickstart["AFR27GH"].geoencode().project(type="mercator")
Visualisation of time series¶
Facilities are provided to plot time series, after a basic cleaning of data (remove NaN values), both with Matplotlib and Altair. The traffic style context offers a convenient first style to customise further.
with plt.style.context("traffic"):
fig, ax = plt.subplots(figsize=(10, 7))
between1920.plot_time(
ax, y=["altitude", "groundspeed"], secondary_y=["groundspeed"]
)
(
quickstart["EZY81GE"].encode("groundspeed")
+ quickstart["EZY743L"].encode("groundspeed")
+ quickstart["AFR27GH"].encode("groundspeed")
)
Sources of data¶
Basic navigational data are embedded in the library, together with parsing facilities for most common sources of information, with a main focus on Europe at the time being.
Airspaces are a key element of aviation: they are regulated by specific rules, whereby navigation is allowed to determined types of aircraft meeting strict requirements. Such volumes, assigned to air traffic controllers to ensure the safety of flights and proper separation between aircraft are most commonly described as a combination of extruded polygons. Flight Information Regions (FIR) are one of the basic form of airspaces.
A non official list of European FIRs, airports, navaids and airways is available in the traffic library (Details here).
from traffic.data import eurofirs
# LISBOA FIR
eurofirs["LPPC"].geoencode()
from traffic.data import airports
airports["AMS"]
EHAM/AMSThe details of airport representations are also available (fetched from OpenStreetMap) in their Matplotlib and Altair representation.
airports["LFBO"].geoencode(runways=True, labels=True)
Intersections can be computed between trajectories and geometries
(airports, airspaces). Flight.intersects() provides a fast boolean
test; Flight.clip() trims the trajectory between the first point of
entry in and last point of exit from the 2D footprint of the geometry.
belevingsvlucht.intersects(airports["EHAM"])
# True
Of course, all these methods can be chained.
(
airports["EHAM"].geoencode(runways=True, labels=True)
+ belevingsvlucht.last(hours=1)
.clip(airports["EHAM"])
.geoencode()
.mark_line(color="crimson")
)
A simple use case¶
The following use case showcases various preprocessing methods that can be chained to select all trajectories landing at Toulouse airport. We will need the coordinates of Toulouse Terminal Maneuvering Area (TMA) which is available in Eurocontrol AIRAC files.
You may not be entitled access to these data but the coordinates of Toulouse TMA are public, so we provide them in this library for the sake of this example.
If you have set the configuration for the AIRAC files (details here), you may uncomment the following cell.
# from traffic.data import nm_airspaces
# lfbo_tma = nm_airspaces["LFBOTMA"]
Since you may not be entitled access to these data and coordinates of Toulouse TMA are public, we provide them in this library for the sake of this example.
from traffic.data.samples import lfbo_tma
lfbo_tma
- 0.0, 65.0
- 65.0, 115.0
- 115.0, 145.0
A first common necessary prepocessing concerns filtering of faulty values, esp. when data comes for a wide network of ADS-B sensors such as the OpenSky Network. A common pattern in such data is spikes in various signals, esp. altitude. Some filtering methods have been developped to take this faulty values out:
hop87dj = quickstart["HOP87DJ"]
# Set a different callsign and identify signals on the visualisation
filtered = hop87dj.filter().assign(callsign="HOP87DJ+")
import altair as alt
# Let's define a common y-scale for both flights
scale = alt.Scale(domain=(0, 40000))
visualisations = [
(flight.encode(alt.Y("altitude", scale=scale)).properties(height=180, width=360))
for flight in [hop87dj, filtered]
]
alt.vconcat(*visualisations)
Let’s select first trajectories intersecting Toulouse TMA, filter signals, then plot the results.
# A progressbar may be convenient...
landing_trajectories = []
for flight in quickstart:
if flight.intersects(lfbo_tma):
filtered = flight.filter()
landing_trajectories.append(filtered)
t_tma = Traffic.from_flights(landing_trajectories)
t_tma
| count | ||
|---|---|---|
| icao24 | callsign | |
| 4ca84d | RYR3YM | 2859 |
| 393320 | AFR27GH | 2770 |
| 393322 | AFR23FK | 2665 |
| 40643a | EZY57FT | 2656 |
| 400cd1 | EZY81GE | 2579 |
| 4ca2c9 | EIN056 | 2526 |
| 44ce6f | BEL7NG | 2182 |
| 406134 | EZY819T | 2050 |
| 4010eb | EZY743L | 1769 |
| 4ca647 | RYR3TL | 1753 |
from traffic.drawing import location
with plt.style.context("traffic"):
fig, ax = plt.subplots(subplot_kw=dict(projection=Lambert93()))
ax.background_patch.set_visible(False)
ax.outline_patch.set_visible(False)
# We may add contours from OpenStreetMap
# (Occitanie is the name of the administrative region)
location("Occitanie").plot(ax, linestyle="dotted")
ax.set_extent("Occitanie")
# Plot the airport, the TMA
airports["LFBO"].plot(ax)
lfbo_tma.plot(ax, linewidth=2, linestyle="dashed")
# and the resulting traffic
t_tma.plot(ax)
There is still one trajectory which does not seem to be coming to Toulouse airport. Also, we actually wanted to select landing trajectories. Let’s only select trajectories coming below 10,000 ft and with an average vertical speed below 1,000 ft/min.
landing_trajectories = []
for flight in quickstart:
if flight.intersects(lfbo_tma):
filtered = flight.filter()
if filtered.min("altitude") < 10_000:
if filtered.mean("vertical_rate") < - 500:
landing_trajectories.append(filtered)
t_tma = Traffic.from_flights(landing_trajectories)
t_tma
| count | ||
|---|---|---|
| icao24 | callsign | |
| 4ca84d | RYR3YM | 2859 |
| 393320 | AFR27GH | 2770 |
| 393322 | AFR23FK | 2665 |
| 400cd1 | EZY81GE | 2579 |
| 44ce6f | BEL7NG | 2182 |
| 4010eb | EZY743L | 1769 |
| 4ca589 | RYR17G | 1643 |
| 405b67 | EZY43UC | 1632 |
| 39b9eb | HOP47FK | 1449 |
from traffic.drawing import location
with plt.style.context("traffic"):
fig, ax = plt.subplots(subplot_kw=dict(projection=Lambert93()))
ax.background_patch.set_visible(False)
ax.outline_patch.set_visible(False)
# We may add contours from OpenStreetMap
# (Occitanie is the name of the administrative region)
location("Occitanie").plot(ax, linestyle="dotted")
ax.set_extent("Occitanie")
# Plot the airport, the TMA
airports["LFBO"].plot(ax)
lfbo_tma.plot(ax, linewidth=2, linestyle="dashed")
# and the resulting traffic
t_tma.plot(ax, alpha=0.5)
for flight in t_tma:
flight_before = flight.before("2017-07-16 20:00")
# Avoid unnecessary noise on the map
if 1000 < flight_before.at().altitude < 20000:
flight_before.plot(ax, alpha=0.5, color="crimson")
flight_before.at().plot(ax, s=20, text_kw=dict(s=flight.callsign))
Lazy iteration¶
Basic operations on Flights define a little language which enables to express programmatically any kind of preprocessing.
The downside with programmatic preprocessing is that it may become unnecessarily complex and nested with loops and conditions to express even basic treatments. As a reference, here is the final code we came to:
# unnecessary list
landing_trajectories = []
for flight in quickstart:
# loop
if flight.intersects(lfbo_tma):
# first condition
filtered = flight.filter()
if filtered.min("altitude") < 10_000:
# second condition
if filtered.mean("vertical_rate") < 1_000:
# third condition
landing_trajectories.append(filtered)
t_tma = Traffic.from_flights(landing_trajectories)
As we define operations on single trajectories, we may also want to express operations, like filtering or intersections on collections of trajectories rather than single ones.
# Traffic.filter() would be
Traffic.from_flights(
flight.filter() for flight in quickstart
)
# Traffic.intersects(airspace) would be
Traffic.from_flights(
flight for flight in quickstart
if flight.intersects(airspace)
)
Such implementation would be very inefficient because Python would constantly start a new iteration for every single operation that is chained. To avoid this, a mechanism of lazy iteration has been implemented:
- Most
Flightmethods returning aFlight, a boolean orNonecan be stacked onTrafficstructures; - When such a method is stacked, it is not evaluated, just pushed for later evaluation;
- The final
.eval()call starts one single iteration and apply all stacked method to everyFlightit can iterate on. - If one of the methods returns
FalseorNone, theFlightis discarded; - If one of the methods returns
True, theFlightis passed as is not the next method.
# A custom grammar can be defined
# here we define conditions for detecting landing trajectories
def landing_trajectory(flight: "Flight") -> bool:
return flight.min("altitude") < 10_000 and flight.mean("vertical_rate") < -500
t_tma = (
quickstart
# non intersecting flights are discarded
.intersects(lfbo_tma)
# intersecting flights are filtered
.filter()
# filtered flights not matching the condition are discarded
.filter_if(landing_trajectory)
# final multiprocessed evaluation (4 cores) through one iteration
.eval(max_workers=4)
)
t_tma
| count | ||
|---|---|---|
| icao24 | callsign | |
| 4ca84d | RYR3YM | 2859 |
| 393320 | AFR27GH | 2770 |
| 393322 | AFR23FK | 2665 |
| 400cd1 | EZY81GE | 2579 |
| 44ce6f | BEL7NG | 2182 |
| 4010eb | EZY743L | 1769 |
| 4ca589 | RYR17G | 1643 |
| 405b67 | EZY43UC | 1632 |
| 39b9eb | HOP47FK | 1449 |