welcome to Fuzzy Tolerance
a technology and software development blog focusing on GIS and open source software. our mission is to foster knowledge sharing, communication, and collaboration with the community.

Archive for the ‘Code’ Category

Using GeoServer, Openlayers, and CQL Filters

Friday, March 5th, 2010

I received a request this week to put Census Questionnaire Centers (QAC) on a web app ASAP. In case you don’t follow the Census’ every move:

Questionnaire Assistance Centers (QACs) are spaces, donated by community partners, where staff from the Local Census Office or the partner organization are available to answer questions about completing the questionnaire, provide special language assistance and answer general questions.

This is a short term thing – the QACs are only open for 4 weeks or so – and the only initial requirement was dumping the points on the map and showing information when you click on them. So I dumped the shape file into Postgres, published the layer with GeoServer, and added it to an app as a vector layer (KML).

For the most part this code was proudly pilfered from one of the OpenLayers samples. Here we’ll add the layer to the map and set some select events. I made the census_qac_layers a global variable so I could get at it later. Note I took out the enormous URL so the page wouldn’t explode, but this was it (it will probably ask you to pull it up in Google Earth).


// QAC Centers for Census
    census_qac_centers = new OpenLayers.Layer.Vector("Census QAC Centers", {
        projection: map.displayProjection,
        strategies: [new OpenLayers.Strategy.Fixed()],
        protocol: new OpenLayers.Protocol.HTTP({
            url: "http://enormous-url",
            format: new OpenLayers.Format.KML({
                extractStyles: true,
                extractAttributes: true
            })
        })
    });
    census_qac_centers.setVisibility(false);
    map.addLayer(census_qac_centers);
    select = new OpenLayers.Control.SelectFeature(census_qac_centers);
    census_qac_centers.events.on({
        "featureselected": onFeatureSelect,
        "featureunselected": onFeatureUnselect,
        "stopClick": true
    });
    map.addControl(select);
    select.activate();

And here are the onFeatureSelect and onFeatureUnselect functions we referenced. They’ll handle creating the popups.

/* Functions for Census QAC Centers */
    function onPopupClose(evt) {
            select.unselectAll();
        }
        function onFeatureSelect(event) {
            var feature = event.feature;
            // Since KML is user-generated, do naive protection against
            // Javascript.
            var content = "<h2>"+feature.attributes.name + "</h2>" + feature.attributes.description;
            if (content.search("<script") != -1) {
                content = "Content contained Javascript! Escaped content below.<br />" + content.replace(/</g, "<");
            }
            popup = new OpenLayers.Popup.FramedCloud("chicken",
                 feature.geometry.getBounds().getCenterLonLat(),
                 new OpenLayers.Size(100,100),
                 content,
                 null, true, onPopupClose);
            feature.popup = popup;
            map.addPopup(popup);
        }
        function onFeatureUnselect(event) {
            var feature = event.feature;
            if(feature.popup) {
                map.removePopup(feature.popup);
                feature.popup.destroy();
                delete feature.popup;
            }
        }

Done and done. Our map now has the pox, and each pustule will pop up the QAC information. I gave the KML output a little styling, which is literally as simple as dropping a couple of files like this in your layer folder (here’s description.ftl):


${partner_na.value} <br />
${street.value} <br />
${phone.value} <br />
Hours: ${hours.value} <br />
<br />
Langauges served: ${language_s.value}

Being at least nominally an agile guy, I (a) release early and often and (b) fully expect changes. So we tossed it out there, looked at it, and decided it didn’t make sense to show all the QAC’s at once. What people would really want to do is show just the locations that support their language. I don’t want to click through 111 locations hunting for a QAC that supports Hindi.

The last thing I wanted to do was make a bunch of separate layers/views/styles for different languages, and an individual QAC can support multiple languages, further complicating the matter.

CQL to the rescue.

CQL, the Contextual Query Language, is a formal language for representing queries to information retrieval systems such as web indexes, bibliographic catalogs and museum collection information. The design objective is that queries be human readable and writable, and that the language be intuitive while maintaining the expressiveness of more complex languages.

CQL was developed by the Library of Congress and OGC used it as a basis for its filter encoding spec (I *think*). GeoServer fully supports the OGC filter encoding sped, but sending it over HTML makes for a horrid mess, as you writing and URL encoding XML. GeoServer also supports plain old CQL, which is quite a bit easier. I needed to add a line to the layer declaration to hold a CQL_Filter parameter.


// QAC Centers for Census
    census_qac_centers = new OpenLayers.Layer.Vector("Census QAC Centers", {
        projection: map.displayProjection,
        strategies: [new OpenLayers.Strategy.Fixed()],
        protocol: new OpenLayers.Protocol.HTTP({
            url: "http://enormous-url",
            'params' : { 'CQL_FILTER' : ''},
            format: new OpenLayers.Format.KML({
                extractStyles: true,
                extractAttributes: true
            })
        })
    });

Next I made a standard HTML radio button list, with the value being the language.


<input type="radio" name="qac_centers" value="English"> English<br />
<input type="radio" name="qac_centers" value="Arabic"> Arabic<br />
<input type="radio" name="qac_centers" value="Chinese"> Chinese<br />
<input type="radio" name="qac_centers" value="French"> French<br />

Etc. Now for some jQuery goodness.


    // Census QAC Stuff
    $("input[name='qac_centers']").attr("checked", false)
    $("input[name='qac_centers']").change(
        function()
        {
            census_qac_centers.setVisibility(true);
            filterval = "language_s like '%" + $("input[name='qac_centers']:checked").val() + "%'";
            census_qac_centers.refresh({
                force: true,
                params: {
                    "CQL_FILTER": filterval
                }
            });
        }
    );

Let’s walk through this. First, we set all the radio buttons to be unchecked (Firefox likes to remember that stuff from session to session). Next we detect whenever a change occurs (i.e. a radio button is checked) and run a function. We set the layer to visible, grab the language value of the checked radio button, and create the CQL filter. It’ll end up looking something like this, where language_s is the field name we’re querying:

language_s like '%Spanish%'

I’m doing a like operation because the field has a comma delimited list of languages it supports. This next bit does the heavy lifting.

census_qac_centers.refresh({
      force: true,
      params: {
                    "CQL_FILTER": filterval
                }
 });

Here we take the CQL_FILTER parameter we added to the layer, and we’re dropping in the CQL we created. The layer is refreshed with the new URL, and only the features that match our CQL filter will come back. A user clicks on German, only the German QAC’s show up.

The CQL Filter option is available on any layer in GeoServer (not just vector), and with the parameter option in OpenLayers it’s very easy to dynamically change the content of a layer. We were able to turn around this request in an extremely short amount of time (a couple of hours, a large portion of which was navel-gazing design) and in a way that’s easy to extract from the application when the QAC’s go away in a month.

You can view it on GeoPortal, second accordion tab on the left.

Posted in Code | 3 Comments »

Using @font-face For Better Typography

Monday, February 22nd, 2010

I’ve been on a big web design kick lately, so I thought I’d do a quick post on using @font-face for custom typography on your web page.

@font-face has been around since CSS2, and the first browser to take advantage of it was Internet Explorer 4(!). It fell out of favor for quite a while, for two reasons:

In today’s world, most browsers (Webkit/Safari/Chrome, Firefox, Opera) support both TrueType and OpenType fonts, so that is your target format. We’ll talk about Internet Explorer in a bit. For this example we’ll use the excellent Fontin font. Drop it somewhere in your web site directory – here we’ll say we stuck them in a folder called fonts.


@font-face {
font-family: "Fontin";
src: local("Fontin"), url(fonts/Fontin-Regular.otf) format("opentype");
}

Here we’re creating our own font family, which we’re naming “Fontin”. Note we didn’t have to call it Fontin just because that’s the name of the font – any old name will do.  Next we’re specifying the source for the font, with two options. The first tells the browser to look on the local machine for a font called Fontin and use it if there is one, saving the user a download. You may not want this if there are a lot of disparate versions of the font you want to use floating around, but Fontin is stable. The second source argument tells the browser to grab the font from the web server.

Specify the Fontin range of fonts (regular, bold, and italic) like so:


@font-face {
font-family: "Fontin";
src: local("Fontin"), url(fonts/Fontin-Regular.otf) format("opentype");
}
@font-face {
font-family: "Fontin";
src: local("Fontin"), url(fonts/Fontin-Bold.otf) format("opentype");
font-weight: bold;
}
@font-face {
font-family: "Fontin";
src: local("Fontin"), url(fonts/Fontin-Italic.otf) format("opentype");
font-style: italic;
}

Now I’ll set up a fontin class that we can add to whatever tags we want to use that font in. Note I give alternate, bail-out fonts in the font family stack just in case.


.fontin {
font-family: Fontin, Arial, Helvetica, serif;
font-size: 14px;
}

Internet Explorer still only supports the EOT format, but that isn’t too big of a problem. First, you’ll need to convert your TTF or OTF font file to an EOT format. There are a lot of ways to do that, but this site does a bangup job, even going directly from OTF to EOT. Then we’ll add another line to our @font-face blocks.


@font-face {
font-family: "Fontin";
src: url("fonts/Fontin-Regular.eot");
src: local("Fontin"), url(fonts/Fontin-Regular.otf) format("opentype");
}
@font-face {
font-family: "Fontin";
src: url("fonts/Fontin-Bold.eot");
src: local("Fontin"), url(fonts/Fontin-Bold.otf) format("opentype");
font-weight: bold;
}
@font-face {
font-family: "Fontin";
src: url("fonts/Fontin-Italic.eot");
src: local("Fontin"), url(fonts/Fontin-Italic.otf) format("opentype");
font-style: italic;
}

Notice the extra src line in each @font-face block pointing to the EOT files? That’s all that is required. Browsers ignore CSS lines they don’t understand, so IE picks up on the EOT and ignores the rest, and vice versa for the other browsers.

Font embedding is a great way to add a typographic flare to your web site, but don’t overdo it. Downloading the font files is time and bandwidth for your users, and with these sorts of things, less is often more.*

*The great lesson of District 9 – a few well done special effects beats a two hour CGI crapstorm. This applies equally well to web design.

Posted in Code | No Comments »

How I Made the Cloud Thing

Monday, February 8th, 2010

I’ve already been asked several times how I did the floating cloud bit in the header, so here it goes. Fair warning: if you were expecting some miraculous bit of coding here you’re going to be sorely disappointed.

First, I’m using jQuery for this, so stick a reference to the Google CDN in your head.


<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js">

</script>

Happy cloud is happy.

Next, you need an image to go blundering about on your web page. If you don’t want it to obscure the page underneath it, you’re going to have to do some creative transparency. Here I made a cloud with a radial shading so the edges are less transparent than the middle. But you could use or make anything you want here. I used and recommend Inkscape if you don’t have a favorite vector graphics editor yet.

We’re going to use the image as a background to a div, which you should stick in whatever container div you want the image to roam around in, probably at the bottom. I stuck it in a div creatively named “page-head”.


<div id="cloud"></div>

Now let’s style that div a bit.


/* moving image */
#cloud {
position: relative;
top: 10px;
left: 0;
width: 299px;
height: 173px;
background: transparent url(img/cloud.png) no-repeat scroll 0 0;
}

Nothing special here. Use the top property to move the image up or down to where you want it. Note I’m setting the left property even though I’m starting it the default (0). This is important as you’ll see in a minute.

Now on to the JavaScript.


$(document).ready(function() {
animateCloud();
});

function animateCloud() {
$("#cloud").animate({
'left':  $("#page-head").width() - $("#cloud").width() - parseInt($("#cloud").css("left")) }, 50000, 'linear', function() {            animateCloud();        });
}

When the page is ready, we call the animateCloud function. From there we’re running a jQuery animation where we change the left property of the could by the container width – the width of the cloud image – the current left property of the image. So when the image is at the starting position, we get the width of the container – the width of the cloud div, since the starting left property of the cloud is 0. This is why we had to specify the position even though it’s 0 – if you didn’t, jQuery would return a NaN for this value. So the image goes across the screen over the course of 50 seconds (50000ms), and when it gets to the end it calls itself again. This time we set the left to 0, since container width – cloud div width – cloud div left property will equal 0. Back it moves, calls itself, ad nauseam.

Told you you’d be disappointed.

Posted in Code | No Comments »

Make Smarter – Apache Lucene and Solr, iPhone Development, GIS Intro

Friday, January 15th, 2010

First up in this month’s Make Smarter is a Location-aware search with Apache Lucene and Solr, courtesy of IBM. Lucene is high performance text search engine, Solr is a search platform using Lucene, and they are both open source projects from Apache. Lucene recently added some basic geospatial functionality. By basic we’re talking point features only (XY) and bounding box and distance searches round out the functionality, but the big thing here is speed and search engine integration. It’s a really information article and shows a glimpse at where spatial tech is going.

Via Ars, Stanford has updated their iPhone development course to cover the latest 3.1 SDK. If you’re looking to develop iPhone apps, this course has received very good reviews and last year’s course was downloaded over 4 million times. The only downer is you’ll need iTunes to get the content at Stanford at iTunes U. Mayhaps I can find a torrent…

I’m a big fan and user of Inkscape, and Máirín Duffy is posting the course contents of an 8 session Inkscape course she’s teaching at a Boston middle school. The 1st and 2nd class notes are out there now, with more to come. It’s a very good introduction Inkscape so far, and since I tend to jump into software in the middle it’s helping me learn a lot of stuff I skipped by. Once you’ve gotten comfortable with Inkscape, check out some of the tutorials at Screencasters.HeathenX.Org.

GIS Lounge highlighted A Gentle Introduction to GIS (PDF), a free 114 page ebook aimed at educators and people new to the field and utilizing QGIS. It’s not only a great introduction to the field and key concepts like rasters, vectors, and topology, by using QGIS it lets newcomers do the things they’re learning about at now cost. It’s a great resource and a high quality book. You can find videos, worksheets, and sample data to complement the book at the QGIS site.

If you’re a Flex developer, InfoQ has a good article on the state of the Flex development ecosystem. It’s more or less a long list of Flex resources, from IDE’s to application frameworks.

Finally, via Between the Poles, all the FOSS4G presentations and videos are now online. Links for presentations and videos can be found in the workshop and session abstracts.

Posted in Code | 2 Comments »

String Parsing for Easier Searches

Monday, December 28th, 2009

Through a lot of testing (and swearing), I’ve found the #1 thing users botch on a web application is text entry. Nothing else comes close. That seemingly innocent text box with the blinking cursor is the primary place your user-programmer dance will end with broken toes and hurt feelings.

One of the things I do to reduce text entry problems is to autocomplete text fields wherever I can. The user can start typing in a street name or an address and an autocomplete list will show up.  With the user only having to get the first few characters of a search string right, my number of didn’t find nothin’ search results goes way down, and the user-programmer dance gets a little better.

The Web is like a dominatrix. Everywhere I turn, I see little buttons ordering me to Submit.

But this still left me with a complexity problem. The search area in one of my apps had a text box for addresses, a text box for places, a text box for a parcel ID, a text box for a street name, two text boxes for intersections, and two drop down lists for different types of government facilities. That’s a whopping 8 form entry fields to perform all of the various searches.

I started thinking about condensing this mess into a single search box. I needed to keep my autocomplete functionality to reduce user headaches, but autocomplete functions have to be sub-second fast to be useful. Otherwise the user outruns them when typing and they don’t do anybody any good. And I couldn’t very well search on everything every time and keep the database calls fast.

Time for some string parsing goodness.

Check these search string snippets out:

Here the user is trying to search for an address, a place, a street name, an intersection, and a parcel ID. As a programmer, what I see is:

In other words, if the string is composed of an integer followed by a string, we can assume it’s an address. If it’s a string with no leading integer, we can assume it’s a place or street name. If it’s a string followed by a &, we know it’s an intersection. If it’s an 8 character string that can be converted to an integer we can assume it’s a parcel ID. So I can parse the search string to narrow down the database query, allowing for fast and targeted autocompletes.

Let’s take a look at how that might look in PHP. We’re looking at the string processing and logic here – the nitty gritty processing code will be specific to your data. First, we’ll get the user input.

$query = preg_replace('/\s\s+/', ' ', trim($_REQUEST['query']));

The regex is just replacing extra spaces in the search string. The trim gets rid of leading or trailing white space. No more regex, I promise.

Now we just need some string testing to see what we’ve got.

if (is_numeric($query)) {
  if (strlen($query) == 8 ) {
    // Process the Parcel ID
  }
  else {
    // Return nothing
  }
}

Here we check to see all we have is a number. If that’s the case, we assume it’s a parcel ID. If it’s 8 characters long, we know it’s a parcel ID and we can process that. Otherwise we ignore it.

If it isn’t a PID, we start looking for everything else. So this will be in an else statement to the original if.

else {
  $query_array = explode(' ', $query);
  $pos = strpos($query, "&amp;amp;amp;amp;");

Here we’re getting an array of elements from the query string. We’re also checking to see if there’s a & character, which tells us to look for an intersection.

if (is_numeric($query_array[0]) ) {
  // find the address
}

If the first string passed is an integer, we’re assuming it’s an address. Remember we’ve already weeded out strings that are nothing but a single integer as parcel ID’s.

else if ($pos != false) {
  // Find possible intersections
}

If it wasn’t an address or a parcel ID and it has a & character in it (the strpos function will return false if the search string isn’t found) we’ll treat it as an intersection, like “Ruth & Something”.

else {
  // get points of interest
}
}

Finally, if it isn’t a parcel ID or an address or an intersection, we’ll assume it’s a point of interest (park, library, etc.), process it as such and close the else loop. We can now condense our 8 form entry monstrosity into a single search box with full autocomplete functionality, with a little help from jQuery on the client side.

Grabs address from .5 million record table in ~18ms. Thank you Postgres.

Some points of interest. Note ~* regex searching is happening.

A little intersection action.

A little help is always appreciated.

Viola – the ubersearch. The one search of Sauron. Or, you know, how the Google does it. Put your search box on top of your page and highlight it so the user’s eyes grab on to it.

There’s only one tricky bit to doing string parsing and categorizing like this: you have to keep an eye on the data. Search fields devoid of form can bite you. What if you have a Sanford & Son point of interest? The & character would make our autocomplete think it’s an intersection. What if you had a point of interest called 101 Main? Our autocomplete logic would have that be an address. So you have to watch your data. But if you can pull it off, your users will thank you for it a thousand times over.

To see an example of this, check out GeoPortal.

Posted in Code | 4 Comments »