Tangram OSM XML Data Source

Thu Aug 25 2016

Overview

Because Mapzen currently supports vector tiles in 3 formats: TopoJSON, GeoJSON and Mapbox vector tiles, it is relatively straightforward to add support for additional data sources. In working on OpenMapKit, I have spent some time thinking about the internals involved with editing and working with OpenStreetMap data. We have been searching long and hard for a new map renderer, and my desire is to do it in a way that OSM data really is a first-class citizen in the mapping library. My thought is that adding OSM XML support is the first step toward creating an OSM editor with Tangram.

OSM XML Tiles

I like that Tangram's data pipeline revolves around the notion of a tile. Although JOSM does not think that way, iD actually does. Individual GET requests are made to the OSM Editing API (0.6) for the bbox of a given tile. It turns out we can do the same thing in Tangram ES.

I built a prototype that takes the approach of directly requesting OSM XML from the OSM Editing API. This XML is then parsed into an OSM data model, similar to what you will see in JOSM and OpenMapKit. We have an in-memory data set for a given tile with access to the OSM elements themselves (nodes, ways, and relations). In addition, we can create a Tangram::Layer from the dataset, allowing tiles to be rendered in a similar manner to vector tiles. We loop through the standalone nodes for points, open ways for lines, and closed ways for polygons.

XML Parser

I'm using pugixml to parse the XML. pugixml is a light-weight, simple and fast XML parser for C++. In addition to fantastic documentation, the library has no dependencies. You can simply add the three source files to the project and include the library's main header file when needed. It is a DOM parser only, but this is sufficient, since the XML being parsed should only comprise of the data within a tile at zoom 16. This usually amounts to less than 500kb.

Basis for Offline OSM Data Store

Direct OSM XML support makes sense online, but the deeper goal for me is to make this work from an offline-data store (SQLite). With that in mind, I'm making the MemoryDataSet a child to an abstract DataSet class. That way we can later create a SQLiteDataSet class that queries a database for OSM objects--an alternative to hitting an online REST endpoint.

Styling and Rendering

There are a couple of challenges with rendering direct OSM data. In particular, current Mapzen data is built around the concept of layers. Different elements are grouped into layers such as roads or buildings. OSM data is flat and instead is given context by tags. Therefore, all of the direct OSM XML data is in a single layer called osmXml. From this layer, we can easily filter elements based on their tag properties and style them accordingly.

Take a look at this scene.yaml. Here you can see how we color different types of buildings based off of their building tag values.

Though it is easy to style and render OSM elements, the topology of the data is different from what you will find in a standard Mapzen vector tile. When using Mapzen vector tiles, you can use Mapzen's house styles as-is, without any modification. When using another data source, it will be up to you to create your own style. For example, if you want to use vector tiles hosted by Mapbox, you will also have to create your own scene.yaml that renders the layers of data that Mapbox provides.

Further Details

Further details about this prototype can be found on the OSM XML Support issue on the Tangram ES Github. Development can be found on the OSM_XML branch of my fork of the repo.

The OSM data model is inspired by OpenMapKit Android and OSMSpatialite.