Dirt 3.0

Imgur

Hold on to your git clone, the Dirt Simple PostGIS HTTP API version 3 has been released. Changes. Big changes. Big wonderful changes.

Fastify

Hapi was the previous web framework for Dirt, and it’s really good. Hapi had a major new release, which included a lot of great things. It also broke Dirt. A lot.

When I’m faced with major refactoring work, I use it as an opportunity to evaluate my tools. That brought me to Fastify.

Fastify is a web framework highly focused on providing the best developer experience with the least overhead and a powerful plugin architecture. It is inspired by Hapi and Express and as far as we know, it is one of the fastest web frameworks in town.

Fastify was created by some of the NodeJS core developers, and aside from being blazing fast (almost twice as fast as Hapi), it is interating quickly, has a great ecosystem, and is very easy to work with. Most importantly, switching to Fastify has led to cleaner, simpler code.

Better Docs

There’s a project README. There’s a README in the routes folder explaining how a route works. There’s a README in the config folder explaining the configurations options. I’ve never made documentation I’m happy with, and this is no exception, but I’m the least unhappy with this documentation as any Github project documentation I’ve done.

Improved GeoJSON and MVT

Both of these routes have been improved. GeoJSON is now created in Postgres via native Postgres JSON tooling rather than being performed in the route itself, leading to much better performance. The MVT route is faster and odd artifacts in polygons at low zoom levels have been eliminated.

Better SQL through Template Strings

The previous release used a SQL-building Javascript library. The more I used it, the more I didn’t like it. My main complaint is that it made the SQL call very hard to grok. Every time there was a problem I’d have to console.log() the SQL to see what was going on.

Now I’m using ES2015 template strings. Things are markedly easier to understand. Compare the old SQL builder code for intersect_point:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
function formatSQL(request) {
  var point = `
    ST_Transform(st_setsrid(st_makepoint(${request.params.x}, ${
    request.params.y
  }), ${request.params.srid}),
    find_srid('', '${request.params.table}', '${request.query.geom_column}'))
  `
  var dwithin = `
    ST_DWithin(${request.params.table}.${request.query.geom_column},
      ${point},
      ${request.query.distance})
  `

  var sql = squel
    .select()
    .from(request.params.table)
    .field(request.query.columns)
    .where(dwithin)
    .where(request.query.filter)
    .limit(request.query.limit)
  if (request.query.sort) {
    sql.order(request.query.sort)
  }
  if (request.query.join) {
    var join = request.query.join.split(';')
    sql.join(join[0], null, join[1])
  }

  return sql.toString()
}

Ug. Good luck debugging that. Now feast your eyes on this beauty:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
const sql = (params, query) => {
  const [x, y, srid] = params.point
    .match(/^((-?\d+\.?\d+)(,-?\d+\.?\d+)(,[0-9]{4}))/)[0]
    .split(',')

  return `
  SELECT 
    ${query.columns}
  
  FROM
    ${params.table}
  
  WHERE
    ST_DWithin(
      ${query.geom_column},      
      ST_Transform(
        st_setsrid(
          st_makepoint(${x}, ${y}), 
          ${srid}
        ),
        (SELECT ST_SRID(${query.geom_column}) FROM ${params.table} LIMIT 1)
      ),
      ${query.distance}
    )
    -- Optional Filter
    ${query.filter ? `AND ${query.filter}` : ''}

  -- Optional sort
  ${query.sort ? `ORDER BY ${query.sort}` : ''}

  -- Optional limit
  ${query.limit ? `LIMIT ${query.limit}` : ''}
  `
}

This is much easier to create, understand, and debug.

Other changes

No more JSONP. The response has a proper CORS header, so plain old JSON works fine. JSONP in 2018 is an anti-pattern.

The Postgres login no longer needs select rights on the geometry_columns view. This removes periodic hair-pulling when a PostGIS upgrade recreates the view and removes old permissions. Dirt now uses a subquery to get the SRID when needed, and the list_layers route essentially runs the SQL call that is used to make the geometry_columns view.

The old search route has been dropped. It had two problems - it was of very limited utility (it was really only useful for an auto-suggest scenario), and it was schema specific, so it didn’t work out of the box. Better idea for this kind of functionality - use the query route and ES2015 Promises. Here’s a general example of that technique using promises and array destructuring:

1
2
3
4
const p1 = fetch('<first query service url>')
const p2 = fetch('<second query service url>')
const [r1, r2] = await Promise.all([p1, p2])
const [suggest1, suggest2] = await Promise.all([r1.json(), r2.json()])