Realtime GTFS Feed Parsing with .NET Core 3

What is GTFS

GTFS, or General Transit Feed Specification is a specification for public transport schedules and associated geographic information.

It started as a Google project, but has changed hands to the MobilityData group. Most public transport agencies implement GTFS (at least in Australia), so its fun to play around with this data to find out what buses and trains are doing.

GTFS currently consists of two parts (the MobilityData group has draft specifications for more):

  • GTFS Static: Contains all the static transport data, including bus stops, scheduled timetables, etc.
  • GTFS Realtime: Contains realtime transport data, in my opinion this is a cooler feed, as it shows you the live location of all buses.

GTFS Realtime

GTFS realtime provides three types of data:

  • Trip updates: This includes delays, cancellations, changed routes
  • Service alerts: Severe weather events that affect services, I live in Australia so this happens more often than you think
  • Vehicle positions: Includes location and congestion levels

However its up to the public transport agency to support these. For example, I’ve found that my local transport agency doesn’t publish the congestion levels of vehicles.

Realtime feeds are served via HTTP as a binary file, and is frequently updated by the server. The feed is protobuf encoded, which can be easily decoded to JSON-like structure if you have the correct protobuf definition.

So the general steps for consuming at GTFS realtime feed are:

  1. Do a GET request to the public transport agency’s server to download the file
  2. Decode the protobuf encoded feed
  3. Process the data, display the vehicles on the map etc
  4. Repeat from Step 1 to keep the data updated

Building a Proof of Concept in .NET Core 3

Now lets build a quick program that can read GTFS realtime feeds, for this I’m going to use .NET Core 3, but these instructions should also work for .NET Core 2 as well.

First we will create a new console application

dotnet new console -o GtfsRealtime
cd GtfsRealtime

Next we will add some required packages:

dotnet add package Google.Protobuf
dotnet add package Grpc.Tools

The Google.Protobuf package contains the common code needed to use protobuf messages. The Grpc.Tools is used to automatically compile protobuf definitions to C# classes. Note that this is designed for compiling protobuf files for use with GRPC, so only protobuf3 files are supported.

Next create a folder in the project called Proto, for storing our protobuf definitions.

mkdir Proto

Configure the csproj file to tell .NET Core to automatically compile the protobuf files

<Project ...>
    <ItemGroup>
        <Protobuf Include="Proto/*.proto" />
    </ItemGroup>
</Project>

Next we need to get the protobuf definition for the GTFS realtime feed, Google provides the feed in proto2, however Grpc.Tools only compiles proto3 files, we need to find a proto3 version of the feed. Luckily I found a project on Github that has converted it to proto3 gtfs-realtime-proto3

We can download it to the Proto folder using the following commands

curl https://raw.githubusercontent.com/briansrepo/gtfs-realtime-proto3/master/gtfs-realtime.proto3 -o Proto/grfs-realtime.proto

Now thats all the setup done, finally we can add some code, put this into your Program.cs

using System;
using System.Net;
using Google.Protobuf;
using Google.Transit.Realtime;

namespace GtfsRealtime
{
    class Program
    {
        static void Main(string[] args)
        {
            // Change this feed url to your local public transport agency
            WebRequest req = HttpWebRequest.Create("https://gtfsrt.api.translink.com.au/feed");
            FeedMessage feed = FeedMessage.Parser.ParseFrom(req.GetResponse().GetResponseStream());

            foreach (FeedEntity entity in feed.Entity)
            {
                Console.WriteLine(entity);
            }
        }
    }
}

Thats it! Pretty cool for a couple lines of code, next lets run it using dotnet run. You should see a heap of JSON printed, now that you have the data you can do whatever you want with it, such as display it on a map.