Time Machine

The source code for Time Machine is on Github.

GDAL is the real hero of this story, handling the various formats (TIFF, JPEG, ECW, and MrSid) and projections and projections that thought they were really one thing when they were really another and all kinds of cruft and craziness that is our aerials. The GDAL commands you need will depend on your data, but here are some things that I found helpful.

You probably won’t be lucky enough to get a single mosaic image, which means you’ll need to build a virtual dataset (VRT).

1
gdalbuildvrt 2009.vrt 2009/*.tif

Occasionally some images in a year would think they were in a different projection, or not have any projection information. Helpful tricks here are the -allow_projeciton_difference option and the -a_srs EPSG:WHATEVER, the latter of which tells your images to take what they think their projection is and stuff it. Sometimes you might also have a hyper-anal person that likes to put tiles in folders in folders in folders. gdal will need a little help with the -input_file_list option, which you can build with a little python.

1
2
3
4
5
6
7
8
9
10
11
12
import os
import sys

rootdir = '.'
file = open("filelist.txt", "w")

for folder, subs, files in os.walk(rootdir):
    files = [ fi for fi in files if fi.endswith(".TIF") ]
    for filename in files:
        file.write(os.path.join(folder, filename).replace(".\\","") + '\n')

file.close()

The giant time suck comes next - creating a single mosaic warped to the One True Projection, EPSG:3857. Best to run before you’re leaving for the day.

1
gdalwarp -overwrite -srcnodata -dstnodata -cutline cutline.shp -crop_to_cutline -multi -co COMPRESS=JPEG -co JPEG_QUALITY=75 -co PHOTOMETRIC=YCBCR -co TILED=YES -tr 1.0 1.0 in.vrt out.tiff

I use a cutline and crop to it as it takes care of nodata values pretty well and I’m not being paid to cache our neighbors. If GDAL whines about JPEG, drop in -ot Byte. If it whines about YCBCR, drop that argument entirely. If you end up with empty data, try dropping the -srcnodata bit. And try a small subset first. Depending on the source data, your machine, and sun spots, this could run for days. Better to know if it’s making a furious ball of nothing as fast as you can. Note I’m doing 1 meter pixels, which is good for zoom level 17 without stretching, and for Mecklenburg makes a single TIFF mosaic with JPEG compression anywhere from 600MB to 2GB. Pull up the mosaic in QGIS and eyeball it before you do anything else.

Next up is building a MBTILES file, which is pretty straight forward.

1
2
gdal_translate -of mbtiles -co TILE_FORMAT=JPEG in.tiff out.mbtiles
gdaladdo -r average out.mbtiles 2 4 8 16 32 64 128 256

The numbers on the end of gdaladdo make the images build out to zoom level 8. Going past 256 doesn’t seem to do anything, and going less stops at zoom level 10. A great addition to gdaladdo would be the ability to specify zoom levels, but that is awfully format/projection specific.