Tangram Offline MBTiles Support

Fri Nov 4 2016

Any time you are in a remote area with your mobile device, you may not have access to the internet. In this situation, your mapping library needs to be able to store it's data offline, on the device. We chose the MBTiles format to store our tiles offline.

MBTiles is a specification that has been maintained since 2011, primarily as a tile storage and transport container. Because it is a widely used method to store tiles, both vector and image, it is only logical to continue using this schema to store vector tiles offline.

The spec itself describes the SQLite database schema that is required for the file to be considered MBTiles. Right now the spec only describes the contents of a tile to be png or jpg. However, there is an open pull request that allows you to specify the MIME type of the data you are packaging. This means that you can now store formats like GeoJSON, TopoJSON, Mapbox Vector Tiles, as well as others, as long as there is a corresponding MIME type. In addition, there will be a compression field in the MBTiles metadata table that declares the codec in which tile contents may be compressed.

In the wild, you can find MBTiles storing vector tiles in several places. The most notable example is osm2vectortiles.org. Mapzen has since merged my MBTiles work and plans on releasing MBTile packs to go along with the functionality in their Metro Extracts sometime during Q4 2016.

MBTiles Support Code Sprint

My code sprint allows you to use MBTiles in two key ways: 1) as a tile store and transport mechanism, and 2) as a cache. You can find this sprint documented as a pull request on the Tangram ES Github. This code has been merged into Hannes Janetzek's branch where he is further refining the concept of data sources and tile sources in the library at large.

Scene Declaration

You can declare a data source to have an mbtiles file path both programmatically and via your scene.yaml.

If you want to add tiles to your cache online, you can put both a url and mbtiles in your data source, and http tile requests will be written to your MBTiles db. If the MBTiles file does not exist, Tangram ES will attempt to create a fresh MBTiles file all set up with the proper schema at that path given.

sources:
    osm:
        type: GeoJSON
        url:  https://vector.mapzen.com/osm/all/{z}/{x}/{y}.json
        mbtiles: /Users/njh/osm-data/mbtiles/winters-mapzen-geojson.mbtiles
        max_zoom: 16
        url_params:
            api_key: vector-tiles-tyHL4AY

You don't need a URL, however. You can have your data source simply fetch tiles from the MBTiles and not request anything from the network.

sources:
    osm:
        type: GeoJSON
        mbtiles: /Users/njh/osm-data/mbtiles/winters-mapzen-geojson.mbtiles
        max_zoom: 16

This works nicely when you have full knowledge and access to your filesystem. On Android, however, you might need to get a file path to your MBTiles programmatically first.

File storageDir = Environment.getExternalStorageDirectory();
File mbtilesFile = new File(storageDir, "tangram-geojson-cache.mbtiles");
map.setMBTiles("osm", mbtilesFile);

This would set the file tangram-geojson-cache.mbtiles to be your MBTiles in the osm datasource. It will use a URL from your scene.yaml to populate your cache just like before (or not). Similarly, Map::setMBTiles is exposed from the native C++ Map interface as well. We will also want to add Objective C bindings.

MBTiles Spec

The MBTiles database schema is built to adhere to Paul Norman's MBTiles 2.0 specification. The metadata table is being populated with the required values declared. In addition to adhering to the spec itself, I'm also creating MBTiles databases with identical schema to the most widely used reference implementation, node-mbtiles. You can see the creation of the database in mbtiles.h. One addition that I'd like to make is that we check the metadata table and make sure that the data format is consistent with what we have declared in our scene.yaml.

SQLiteCpp

I'm quite happy with the use of SQLiteCpp C++ SQLite3 wrapper library. It integrates easily with the CMake build process. It works without any issue on Mac OS X and Android, and it's usage is part of the core library. One thing that sets this MBTiles implementation apart from the others is that it is done in the core C++ library. We do not re-invent the wheel and re-implement for Android, iOS, etc.

MD5 Checksum Hash

One clever feature found in node-mbtiles that is not mentioned in the spec is that the underlying database schema has a map and images table that are joined to get you your tile data. The key that joins these tables is an MD5 checksum of the actual tile data. If you have many tiles with duplicate or empty data structures, rather than having a repeat entry in the images table, the checksum will join you to a single entry in images to all of the tiles pointing to that same data.

Rather than integrating a big encryption library like OpenSSL, I just wanted the MD5 function. So, after a bit of searching, I found a C++ hash library that was built to be portable and easy to integrate. I was able to bring in only the MD5 function, therefore saving us from adding extra clutter to the code base. I have a readme that goes into greater detail. I did manually test the hash values with several other MD5 generators and found that the output values matched with the same input.

MBTiles Tile Task

You'll notice that reads and writes to the database are being done inside of an MBTilesTileTask. This class is a subclass of DownloadTileTask, and it is run a tile worker thread, so we can do our blocking call to the database without trouble.

This will work a bit differently when the MBTiles feature is finally released, because the general functionality of tile tasks is going through some changes.

Example App

Stay tuned to the progress of this feature. Once Mapzen releases Metro Extract MBTiles packs, I will create an Android app that lets you download and select an MBTiles file.

Offline MBTiles Support