- [Instructor] Let's look at a complex,
real-world application, and how to approach it
with a server-less, microservices solution.
Throughout this chapter, I'll discuss a transportation
inside dashboard that might be used behind the scenes
of a public transit agency in a metropolitan area.
In this case, I'll talk about a rail system.
But it could also be buses, or ferries, or trams.
In the real world, timetables can be published
at random times throughout the day
and subsequent publications take precedence
over previous versions.
The system will keep track of where
all the trains are located
and whether or not they're on time.
It will show punctuality metrics in real time
and will characterize delays when they occur,
so that train controllers can understand
if a delay pattern will likely impact other trips
on the network.
Clearly, there are a lot of moving pieces.
Not only on the rails,
but also in the system I'll illustrate.
Before we begin, we have to look at what data is available
to approach the problem.
On any given day, there are timetables for each
of the various train lines.
The transit infrastructure provides the identification
and the location of each train once it beings its trip.
There is semi-static, geospatial data for the tracks.
Calculating punctuality metrics for a particular train
requires the relevant timetable and the actual times
that the train has arrived at and departed from
each station on the route.
What are the top level components of the system?
There are two primary data pipelines.
One for timetables
and a second for vehicle position data.
First, let's understand the steps that we need to follow
to process this data.
Call the timetable service if there are new timetables,
then download them.
Process each timetable package and insert it
into our database.
The next data pipeline will process
vehicle position reports.
This data is updated in near real-time
for each one of the trains in operation.
So, there may be hundreds of trains in operation
on the network at any given time.
There are few complications that arise
that we'll discuss a little later.
The steps we'll follow to process the data are:
call the vehicle position report service
run by the transit agency
and serialize the data to JSON for further processing,
process the JSON data and insert it into the database.
We'll need to determine when each train
reaches a station stop.
Calculate the metrics.
Can we characterize a delay?
Cache the results where we can.
How we build the delay characterization model
in this model, remember, the type of delay matters,
so we'll want to know what kind it is.