Talk about Deal

Nowadays Twitter is changing how the web is used. In fact under the pressure of the twitter’s FireHose even Google Search is starting to show transient informations, i.e. the tweets (cfr. Google Tweets and updates and search). As everybody knows a tweet is a short text message (less than 140 characters) and it is intended to deliver ’short living’ messages:

Furthermore, on Nov. 19 2009, Twitter turned on geo-location functionality and started to give the possibility to external applications to provide geo-tagging features to end user: “We are really looking forward to seeing the creative uses emerge from the developer community,” (Ryan Sarver , twitter official blog) and on Dec. 23 2009 Twitter, on the official blog, announced a major new step into the location-aware future, i.e. The acquisition of Mixer Labs, creators of GeoAPI.

This new opportunity is increasing enormously the number of geo-located tweets that can be easily accessed by the twitter search API and give rise to tons of “funny games” with tweets.

The basic one is to show where tweets are coming from. Our today’s pet project is exactly showing tweets on a Google map (almost) continuously. Test it here.

Getting Tweets

Twitter exposes its data via a combination of at least two Application Programming Interfaces (APIs) but for our purpose the simple and yet powerful Search API (rigurously the search method of the RESTful Search API) is largely sufficient. What we need to sample tweets in a given region, say 10km around my home, is to access the URL:

where, of course, [LAT] and [LNG] have to be replaced with the actual coordinates of my home. The call do not require authentication and, as explained in the official documentation there are a lot of criteria you can use to get a specific set of tweets (for example include q=XYZ to get the tweets that contain the word XYZ). Notably the shown request will bring the selected tweets as a JSON (www.json.org) object so trivially usable in javascript.

Unfortunately the direct call to the url in javascript does not work, at least if you are not able to host your page on twitter.com domain. In fact the so called Same Origin Policy (SOP) that is in some sense the heart of the security model of web browsers will not let you to execute programmatically a AJAX (XMLHttp,actually) request to a page whose domain is not the one the page has been loaded from, i.e. a javascript script hosted on www.my_home_site.com cannot request content from www.a_different_domain.com’s site. SOP is intended to limit the possibility of ‘malicious code injection’ and cannot be avoided, but can be in some sense mitigated with some tricks.

The simpler approach to circunvent SOP is named JSONP and is simple to apply but requires some help from the server, essentially the response must contain a function call that our application will execute on return (and this is exactly cone injection !): essentially to return the json object {'a' : true, 'b' : false} the server should include an enclosing a function call:

The good news is that the twitter search method has exactly this feature and just appending to the request &callback= function_my_app_implements we’ll get back the desired JSONP response. So we can write down a simple Jacascript function as:

loadFromTwitter = function(lat,lng,r) {
   var url='http://search.twitter.com/search.json?geocode='
              +lat+'%2C'
              +lng+'%2C'+r
              +'km&callback=manage_response    var script = document.createElement('script');
   script.setAttribute('src', url);
   document.getElementsByTagName('head')[0].appendChild(script);
}

and when the request finishes manage_response will be called with as argument an object containing an array of tweets (unfortunately no error control here, but there are different techniques with better control, even if less straightforward); just to show how manage_response may look like:

manage_response = function(tl) {
    var results = tl.results;
    for(var i =0; i< results.length;i++) {
         do_something_meaningful_with_a_tweet(tw);
    }
}

Putting Tweets on the Google map

Once the tweets reach our javascript application, showing them on the map is a simple game using the Google javascript map api (take a look at the Google Map documentation). Each tweet has a lot of informations (for example: from_user, text, source, …) but most notably for our purposes the tweets returned with a request containing geocode=... param has a location element, so is straightforward to write down a javascript function in order to put the marker on the map:

function createMarker(tweet) {
   var location = tweet.location;
   var point = getPointFromLocation(location);
   var marker = new GMarker(point,{icon:twitterIcon});
   GEvent.addListener(marker, function() {
        var src = '
<div class="tweet_text">' +
         ' <img src="' + tw.profile_image_url +' " alt="" width="30px" height="30px" />' +
         '<span class="twitter_marker">' + tweet.text +
         '</span></div>
';
          marker.openInfoWindowHtml(str);
    });
  return marker;
}

where the added listener opens the GInfoWindow on the marker showing the current tweet; for instance the profile image of the author and the text of the tweet in the sample code. Once the marker is defined, we can put it on the map using the addOverlay(overlay:GOverlay map’s method, where the marker is the overlay we are going to add in our map.

As the Twitter official documentation says: “The location is preferentially taking from the Geotagging API, but will fall back to their Twitter profile”. This means that a tweet location may be given in a few of different formats, for example using coordinates (latitude and longitude) or by address. In the above code the function getPointFromLocation(location) has so to process the location field of the tweet; in the better case, it simply splits the coordinates and returns the LatLng point, while if the location is an address the function has to use the GClientGeocoder in order to request to Google servers coordinates for the user specified location text. Actually the above snippet is too simple to handle geolocation. Indeed as any server request geolocation (namely GclientGeocoder.getLatLng) is asyncronous so the code has to be refactored such that it searches (we do it using regular expressions) for latitude and longitude in the location field and if coordinates are found we create the marker directly, if there are not coordinates ready to be used it has to call the geocoder and eventually, upon successful return, create the marker:

….
geocoder.getLatLng(tweet.location, function(point) {
   if(point) {
      addOverlay(createMarker(tweet,point));
    }
});
…..

Finally, the actual code has to be a bit smarter because geolocation is a time consuming operation and many tweets have the same location, so caching is a must and finally because many tweets may and up at the same coordinates so the infowindow content has to be composed by multiple tweets (and pagination become unavoidable too), but this is out of our intent here.

Continuously

The tweet stream is continuous, so an application dealing with twitter have to be “fresh” in order to show what is appenning in the world. For this purpose javascript setInterval function results in an obvious choice giving us the possibility to call periodically loadFromTwitter; the latitude, longitude and radius are chosen on the basis of the current view of the shown map:

setInterval("periodicLoader()",1000);
periodicLoader = function() {
     var c = map.getCenter();
     var bnd = map.getBounds();
     var sw = bnd.getSouthWest();
     var ne = bnd.getNorthEast();
     var radius = Math.floor( (sw.distanceFrom(ne)/2.0)*0.8 );
     loadFromTwitter(c.lat(),c.lng(), radius);
}

being map our global GMap instance.

Let me note that Tweetter provides an experimental stream access to tweets (a sampling of the firehose right now) that may be the right coice for receiving updates on the map continuously but the streaming api has not anonymous access and handling authentication in the sample code is going to complicate things a lot; on the other hand i think i will not resist the temptation of trying, ASAP, to use the stream, being this a fantastic project where to use HTML5 web-workers :).

This work is realized jointly with Alberto Mancini.