Quickstart

The motivation for this page/notebook is to take the reader through all basic functionalities of the traffic library. We will cover:

  1. a basic introduction about Flight and Traffic structures;
  2. how to produce visualisations of trajectory data;
  3. how to access basic sources of data;
  4. a simple use case to select trajectories landing at Toulouse airport;
  5. an introduction to declarative descriptions of data preprocessing through lazy iteration.

This page is also available as a notebook which can be downloaded and executed locally; or loaded and executed in Google Colab.

Basic introduction

The traffic library provides natural methods and attributes that can be applied on trajectories and collection of trajectories, all represented as pandas DataFrames.

The Flight structure

Flight is the core class offering representations, methods and attributes to single trajectories. A comprehensive description of the API is available here.

Sample trajectories are provided in the library: belevingsvlucht is one of them, context is explained here.

from traffic.data.samples import belevingsvlucht

belevingsvlucht
Flight TRA051
  • aircraft: 484506 / PH-HZO (B738)
  • from: 2018-05-30 15:21:38+00:00
  • to: 2018-05-30 20:22:56+00:00

Among available attributes, you may want to access:

  • its callsign (the identifier of the flight displayed on ATC screens);
  • its transponder unique identification number (icao24);
  • its registration number (tail number);
  • its typecode (i.e. the model of aircraft).
(
    belevingsvlucht.callsign,
    belevingsvlucht.icao24,
    belevingsvlucht.registration,
    belevingsvlucht.typecode,
)

# ('TRA051', '484506', 'PH-HZO', 'B738')

Methods are provided to select relevant parts of the flight, e.g. based on timestamps.

The start and stop attributes refer to the timestamps of the first and last recorded samples. Note that all timestamps are by default set to universal time (UTC) as it is common practice in aviation.

(belevingsvlucht.start, belevingsvlucht.stop)

# (Timestamp('2018-05-30 15:21:38+0000', tz='UTC'),
#  Timestamp('2018-05-30 20:22:56+0000', tz='UTC'))
first30 = belevingsvlucht.first(minutes=30)
after19 = belevingsvlucht.after("2018-05-30 19:00", strict=False)

# Notice the "strict" comparison (>) vs. "or equal" comparison (>=)
print(f"between {first30.start:%H:%M:%S} and {first30.stop:%H:%M:%S}")
print(f"between {after19.start:%H:%M:%S} and {after19.stop:%H:%M:%S}")

# between 15:21:38 and 15:51:37
# between 19:00:00 and 20:22:56
between1920 = belevingsvlucht.between(
    "2018-05-30 19:00", "2018-05-30 20:00"
)
between1920
Flight TRA051
  • aircraft: 484506 / PH-HZO (B738)
  • from: 2018-05-30 19:00:01+00:00
  • to: 2018-05-30 19:59:59+00:00

The underlying dataframe is always accessible.

between1920.data.head()
timestamp icao24 latitude longitude groundspeed track vertical_rate callsign altitude
11750 2018-05-30 19:00:01+00:00 484506 52.839973 5.793947 290 52 -1664 TRA051 8233
11751 2018-05-30 19:00:02+00:00 484506 52.840747 5.795680 290 52 -1664 TRA051 8200
11752 2018-05-30 19:00:03+00:00 484506 52.841812 5.797501 290 52 -1664 TRA051 8200
11753 2018-05-30 19:00:04+00:00 484506 52.842609 5.799133 290 52 -1599 TRA051 8149
11754 2018-05-30 19:00:05+00:00 484506 52.843277 5.801010 289 52 -1599 TRA051 8125

The Traffic structure

Traffic is the core class to represent collections of trajectories, which are all flattened in the same pandas DataFrame. A comprehensive description of the API is available here.

We will demonstrate here with a sample of ADS-B data from the OpenSky Network.

The basic representation of a Traffic object is a summary view of the data: the structure tries to infer how to separate trajectories in the data structure based on customizable heuristics, and returns a number of sample points for each trajectory.

from traffic.data.samples import quickstart

quickstart
Traffic with 397 identifiers
count
icao24 callsign
4ca84d RYR3YM 2859
393320 AFR27GH 2770
505c98 RAM667 2752
3944ef HOP87DJ 2731
4ca574 IBK5111 2706
393322 AFR23FK 2665
40643a EZY57FT 2656
394c18 AFR140W 2613
344692 VLG2972 2599
400cd1 EZY81GE 2579

Traffic offers the ability to index and iterate on all flights contained in the structure. Traffic will use either:

  • a combination of timestamp, icao24 (aircraft identifier) and callsign (mission identifier); or
  • a customizable flight identifier (flight_id);

as a way to separate and identify flights.

Indexation will be made on either of icao24, callsign (or flight_id if available).

quickstart["AFR27GH"]  # on callsign
quickstart["393320"]  # on icao24
Flight AFR27GH
  • aircraft: 393320 / F-GMZA (A321)
  • from: 2017-07-16 19:30:00+00:00
  • to: 2017-07-16 20:16:10+00:00

A subset of trajectories can also be selected if a list is passed an index:

quickstart[["AFR27GH", "HOP87DJ"]]
Traffic with 2 identifiers
count
icao24 callsign
393320 AFR27GH 2770
3944ef HOP87DJ 2731

In many cases, flight_id are more convenient to access specific flights yielded by iteration. We may construct custom flight_id:

from traffic.core import Traffic

quickstart_id = Traffic.from_flights(
    flight.assign(flight_id=f"{flight.callsign}_{i:03}")
    for i, flight in enumerate(quickstart)
)
quickstart_id
Traffic with 379 identifiers
count
flight_id
RYR3YM_343 2859
AFR27GH_046 2770
RAM667_373 2752
HOP87DJ_055 2731
IBK5111_316 2706
AFR23FK_048 2665
EZY57FT_172 2656
AFR140W_064 2613
VLG2972_036 2599
EZY81GE_149 2579
or use the available .assign_id() method, which is implemented exactly that way.
(We will explain eval() further below)
quickstart.assign_id().eval()
Traffic with 379 identifiers
count
flight_id
RYR3YM_343 2859
AFR27GH_046 2770
RAM667_373 2752
HOP87DJ_055 2731
IBK5111_316 2706
AFR23FK_048 2665
EZY57FT_172 2656
AFR140W_064 2613
VLG2972_036 2599
EZY81GE_149 2579

Saving and loading data

Some processing operations are computationally expensive and time consuming. Therefore, it may be relevant to store intermediate results in files for sharing and reusing purposes.

One option is to store Traffic and Flight underlying DataFrames in pickle format. Details about storage formats are presented here.

quickstart_id.to_pickle("quickstart_id.pkl")
from traffic.core import Traffic

# load from file again
quickstart_id = Traffic.from_file("quickstart_id.pkl")

Visualisation of data

traffic offers facilities to leverage the power of common visualisation renderers including Cartopy, a map plotting library built around Matplotlib, and Altair.

More visualisation renderers such as Leaflet are available as plugins.

Visualisation of trajectories

When you choose to plot trajectories on a map, you have to make a choice concerning how to represent points at the surface of a sphere (more precisely, an oblate spheroid) on a 2D plane. This transformation is called a projection.

The choice of the right projection depends on the data. The most basic projection (sometimes wrongly referred to as no projection) is the PlateCarree(), when you plot latitude on the y-axis and longitude on the x-axis. The famous Mercator() projection distorts the latitude so as lines with constant bearing appear as straight lines. Conformal projections are also convenient when plotting smaller areas (countries) as they preserve distances (locally).

Many countries define official projections to produce maps of their territory. In general, they fall either in the conformal or in the Transverse Mercator category. Lambert93() projection is defined over France, GaussKruger() over Germany, Amersfoort() over the Netherlands, OSGB() over the British Islands, etc.

When plotting trajectories over Western Europe, EuroPP() is a decent choice.

from traffic.data.samples import airbus_tree
%matplotlib inline
import matplotlib.pyplot as plt

from traffic.core.projection import Amersfoort, GaussKruger, Lambert93, EuroPP
from traffic.drawing import countries

with plt.style.context("traffic"):
    fig = plt.figure()

    # Choose the projection type
    ax0 = fig.add_subplot(221, projection=EuroPP())
    ax1 = fig.add_subplot(222, projection=Lambert93())
    ax2 = fig.add_subplot(223, projection=Amersfoort())
    ax3 = fig.add_subplot(224, projection=GaussKruger())

    for ax in [ax0, ax1, ax2, ax3]:
        ax.add_feature(countries())
        # Maximum extent for the map
        ax.set_global()
        # Remove border and set transparency for background
        ax.outline_patch.set_visible(False)
        ax.background_patch.set_visible(False)

    # Flight.plot returns the result from Matplotlib as is
    # Here we catch it to reuse the color of each trajectory
    ret, *_ = quickstart["AFR27GH"].plot(ax0)
    quickstart["AFR27GH"].plot(
        ax1, color=ret.get_color(), linewidth=2
    )

    ret, *_ = belevingsvlucht.plot(ax0)
    belevingsvlucht.plot(
        ax2, color=ret.get_color(), linewidth=2
    )

    ret, *_ = airbus_tree.plot(ax0)
    airbus_tree.plot(
        ax3, color=ret.get_color(), linewidth=2
    )

    # We reduce here the extent of the EuroPP() map
    # between 8°W and 18°E, and 40°N and 60°N
    ax0.set_extent((-8, 18, 40, 60))

    params = dict(fontname="Ubuntu", fontsize=18, pad=12)

    ax0.set_title("EuroPP()", **params)
    ax1.set_title("Lambert93()", **params)
    ax2.set_title("Amersfoort()", **params)
    ax3.set_title("GaussKruger()", **params)

    fig.tight_layout()
_images/quickstart_map.png

Altair API is not very mature yet with geographical data, but basic visualisations are possible.

# Mercator projection is the default one with Altair
quickstart["AFR27GH"].geoencode().project(type="mercator")
_images/quickstart_trajectory.png

Visualisation of time series

Facilities are provided to plot time series, after a basic cleaning of data (remove NaN values), both with Matplotlib and Altair. The traffic style context offers a convenient first style to customise further.

with plt.style.context("traffic"):
    fig, ax = plt.subplots(figsize=(10, 7))
    between1920.plot_time(
        ax, y=["altitude", "groundspeed"], secondary_y=["groundspeed"]
    )
_images/quickstart_plottime.png
(
    quickstart["EZY81GE"].encode("groundspeed")
    + quickstart["EZY743L"].encode("groundspeed")
    + quickstart["AFR27GH"].encode("groundspeed")
)
_images/quickstart_altair.png

Sources of data

Basic navigational data are embedded in the library, together with parsing facilities for most common sources of information, with a main focus on Europe at the time being.

Airspaces are a key element of aviation: they are regulated by specific rules, whereby navigation is allowed to determined types of aircraft meeting strict requirements. Such volumes, assigned to air traffic controllers to ensure the safety of flights and proper separation between aircraft are most commonly described as a combination of extruded polygons. Flight Information Regions (FIR) are one of the basic form of airspaces.

A non official list of European FIRs, airports, navaids and airways is available in the traffic library (Details here).

from traffic.data import eurofirs

# LISBOA FIR
eurofirs["LPPC"].geoencode()
_images/quickstart_lisboa.png
from traffic.data import airports

airports["AMS"]
Amsterdam Schiphol Airport (Netherlands) EHAM/AMS

The details of airport representations are also available (fetched from OpenStreetMap) in their Matplotlib and Altair representation.

airports["LFBO"].geoencode(runways=True, labels=True)
_images/quickstart_lfbo.png

Intersections can be computed between trajectories and geometries (airports, airspaces). Flight.intersects() provides a fast boolean test; Flight.clip() trims the trajectory between the first point of entry in and last point of exit from the 2D footprint of the geometry.

belevingsvlucht.intersects(airports["EHAM"])
# True

Of course, all these methods can be chained.

(
    airports["EHAM"].geoencode(runways=True, labels=True)
    + belevingsvlucht.last(hours=1)
    .clip(airports["EHAM"])
    .geoencode()
    .mark_line(color="crimson")
)
_images/quickstart_eham.png

A simple use case

The following use case showcases various preprocessing methods that can be chained to select all trajectories landing at Toulouse airport. We will need the coordinates of Toulouse Terminal Maneuvering Area (TMA) which is available in Eurocontrol AIRAC files.

You may not be entitled access to these data but the coordinates of Toulouse TMA are public, so we provide them in this library for the sake of this example.

If you have set the configuration for the AIRAC files (details here), you may uncomment the following cell.

# from traffic.data import nm_airspaces
# lfbo_tma = nm_airspaces["LFBOTMA"]

Since you may not be entitled access to these data and coordinates of Toulouse TMA are public, we provide them in this library for the sake of this example.

from traffic.data.samples import lfbo_tma

lfbo_tma
SIV TOULOUSE [SIV TOULOUSE] (AUA)
  • 0.0, 65.0
  • 65.0, 115.0
  • 115.0, 145.0

A first common necessary prepocessing concerns filtering of faulty values, esp. when data comes for a wide network of ADS-B sensors such as the OpenSky Network. A common pattern in such data is spikes in various signals, esp. altitude. Some filtering methods have been developped to take this faulty values out:

hop87dj = quickstart["HOP87DJ"]
# Set a different callsign and identify signals on the visualisation
filtered = hop87dj.filter().assign(callsign="HOP87DJ+")
import altair as alt

# Let's define a common y-scale for both flights
scale = alt.Scale(domain=(0, 40000))

visualisations = [
    (flight.encode(alt.Y("altitude", scale=scale)).properties(height=180, width=360))
    for flight in [hop87dj, filtered]
]

alt.vconcat(*visualisations)
_images/quickstart_filter.png

Let’s select first trajectories intersecting Toulouse TMA, filter signals, then plot the results.

# A progressbar may be convenient...
landing_trajectories = []

for flight in quickstart:
    if flight.intersects(lfbo_tma):
        filtered = flight.filter()
        landing_trajectories.append(filtered)

t_tma = Traffic.from_flights(landing_trajectories)
t_tma
Traffic with 17 identifiers
count
icao24 callsign
4ca84d RYR3YM 2859
393320 AFR27GH 2770
393322 AFR23FK 2665
40643a EZY57FT 2656
400cd1 EZY81GE 2579
4ca2c9 EIN056 2526
44ce6f BEL7NG 2182
406134 EZY819T 2050
4010eb EZY743L 1769
4ca647 RYR3TL 1753
from traffic.drawing import location

with plt.style.context("traffic"):
    fig, ax = plt.subplots(subplot_kw=dict(projection=Lambert93()))
    ax.background_patch.set_visible(False)
    ax.outline_patch.set_visible(False)

    # We may add contours from OpenStreetMap
    # (Occitanie is the name of the administrative region)
    location("Occitanie").plot(ax, linestyle="dotted")
    ax.set_extent("Occitanie")

    # Plot the airport, the TMA
    airports["LFBO"].plot(ax)
    lfbo_tma.plot(ax, linewidth=2, linestyle="dashed")

    # and the resulting traffic
    t_tma.plot(ax)
_images/quickstart_tma.png

There is still one trajectory which does not seem to be coming to Toulouse airport. Also, we actually wanted to select landing trajectories. Let’s only select trajectories coming below 10,000 ft and with an average vertical speed below 1,000 ft/min.

landing_trajectories = []

for flight in quickstart:
    if flight.intersects(lfbo_tma):
        filtered = flight.filter()
        if filtered.min("altitude") < 10_000:
            if filtered.mean("vertical_rate") < - 500:
                landing_trajectories.append(filtered)

t_tma = Traffic.from_flights(landing_trajectories)
t_tma
Traffic with 9 identifiers
count
icao24 callsign
4ca84d RYR3YM 2859
393320 AFR27GH 2770
393322 AFR23FK 2665
400cd1 EZY81GE 2579
44ce6f BEL7NG 2182
4010eb EZY743L 1769
4ca589 RYR17G 1643
405b67 EZY43UC 1632
39b9eb HOP47FK 1449
from traffic.drawing import location

with plt.style.context("traffic"):
    fig, ax = plt.subplots(subplot_kw=dict(projection=Lambert93()))
    ax.background_patch.set_visible(False)
    ax.outline_patch.set_visible(False)

    # We may add contours from OpenStreetMap
    # (Occitanie is the name of the administrative region)
    location("Occitanie").plot(ax, linestyle="dotted")
    ax.set_extent("Occitanie")

    # Plot the airport, the TMA
    airports["LFBO"].plot(ax)
    lfbo_tma.plot(ax, linewidth=2, linestyle="dashed")

    # and the resulting traffic
    t_tma.plot(ax, alpha=0.5)

    for flight in t_tma:
        flight_before = flight.before("2017-07-16 20:00")

        # Avoid unnecessary noise on the map
        if 1000 < flight_before.at().altitude < 20000:

            flight_before.plot(ax, alpha=0.5, color="crimson")
            flight_before.at().plot(ax, s=20, text_kw=dict(s=flight.callsign))
_images/quickstart_final.png

Lazy iteration

Basic operations on Flights define a little language which enables to express programmatically any kind of preprocessing.

The downside with programmatic preprocessing is that it may become unnecessarily complex and nested with loops and conditions to express even basic treatments. As a reference, here is the final code we came to:

# unnecessary list
landing_trajectories = []

for flight in quickstart:
    # loop
    if flight.intersects(lfbo_tma):
        # first condition
        filtered = flight.filter()
        if filtered.min("altitude") < 10_000:
            # second condition
            if filtered.mean("vertical_rate") < 1_000:
                # third condition
                landing_trajectories.append(filtered)

t_tma = Traffic.from_flights(landing_trajectories)

As we define operations on single trajectories, we may also want to express operations, like filtering or intersections on collections of trajectories rather than single ones.

# Traffic.filter() would be
Traffic.from_flights(
    flight.filter() for flight in quickstart
)

# Traffic.intersects(airspace) would be
Traffic.from_flights(
    flight for flight in quickstart
    if flight.intersects(airspace)
)

Such implementation would be very inefficient because Python would constantly start a new iteration for every single operation that is chained. To avoid this, a mechanism of lazy iteration has been implemented:

  • Most Flight methods returning a Flight, a boolean or None can be stacked on Traffic structures;
  • When such a method is stacked, it is not evaluated, just pushed for later evaluation;
  • The final .eval() call starts one single iteration and apply all stacked method to every Flight it can iterate on.
  • If one of the methods returns False or None, the Flight is discarded;
  • If one of the methods returns True, the Flight is passed as is not the next method.
# A custom grammar can be defined
# here we define conditions for detecting landing trajectories

def landing_trajectory(flight: "Flight") -> bool:
    return flight.min("altitude") < 10_000 and flight.mean("vertical_rate") < -500


t_tma = (
    quickstart
    # non intersecting flights are discarded
    .intersects(lfbo_tma)
    # intersecting flights are filtered
    .filter()
    # filtered flights not matching the condition are discarded
    .filter_if(landing_trajectory)
    # final multiprocessed evaluation (4 cores) through one iteration
    .eval(max_workers=4)
)
t_tma
Traffic with 9 identifiers
count
icao24 callsign
4ca84d RYR3YM 2859
393320 AFR27GH 2770
393322 AFR23FK 2665
400cd1 EZY81GE 2579
44ce6f BEL7NG 2182
4010eb EZY743L 1769
4ca589 RYR17G 1643
405b67 EZY43UC 1632
39b9eb HOP47FK 1449