Viewing missing methods/software, part V: Processing

In my previous post, I was  bemoaning the slow speed of the visualization chart using Google Maps. Looking around in the Google Maps group, it seemed that there wasn't a good way to speed up the drawing of many markers, except for drawing them in another program first and storing them on image tiles. Since I was going to have to draw markers in a different program anyway, I decided to try using such a program by itself without involving Google Maps. First, I thought to create a static image using GD with PHP and then do an image map on top of that with javascript, but GD wasn't working on the NESCent server (this was fixed in just a couple of hours, as is common with NESCent's great tech support). While it was being fixed, I decided to learn Processing — I've been impressed by the quality of some of the diagrams it creates, and having a dynamic diagram might be more useful for users. It has good documentation and can parse XML, so within a few hours, I learned enough to be able to create the diagram below. 

Following advice from the TreeTapper design consultant (my wife), I made the nodes smaller and changed the color slightly, but otherwise it is the same as the Google diagram (though I don't have popups or mouseovers working yet), and is much faster to create. It does require users to have Java installed in their browsers, but might allow cool features later on.

Viewing missing methods/software, part IV: Sort of working

Well, I sort of have the visualization working. Users can choose which elements to graph (on Firefox) by dragging boxes to move them above or below a plotting line, and they can choose to limit to plotting only one of many options. See below.

On clicking the  update chart button, a tree is plotted using the Google Maps API, with branches colored based on whether they have no methods or software, methods but not software, or both methods and software [I originally had it update the map automatically on any change, but this is too slow]. Users can click on nodes to get an info window showing the choices made going from the tree root to the tips and links to any relevant software or methods. Just mousing over a node tells a user what option that node represents (i.e., "Treetype: Unrooted, polytomies, incomplete tree"). If a user has chosen just one setting for an option (as for criterion, below), the edge leading to that node is shorter and light gray edges are shown to indicate the options not examined.  See below (click to make larger).

The end product is beautiful and information-rich. It's also VERY slow. It takes literally minutes to generate the plot, and then >10 seconds between clicking on a node and getting the info window. Zooming in or out or moving the plot also take an agonizingly slow time. Getting info from the script that talks to the database to make code for the map takes a while, about 30 seconds, but the real slowdown comes when drawing the map. Google maps are slow with many markers and polylines: the map above has 221 markers and even more polylines, with the three circle polylines having 360 points each (using fewer makes a plot that's too rough). I'll have to decide what to do. I'd stayed away from Processing due to usability concerns, but it seems Google Maps isn't so great, either. I've read a little about generating image tiles rather than markers to speed up the map -- I'll look into this and other options. 

Viewing missing methods/software, part II

It appears the strategy of using Google Maps with YUI display of items to add to the map will basically work. Rather than doing drag and drop between YUI datatables, I'm using just the drag and drop YUI code on a list of options, each with the possibility of limiting it to one item (for example, one can first organize by optimality criterion, deciding to show all or just likelihood). The interface is based on the YUI example, but with just one list, with one element of a different color so that options placed above this element appear on the tree while ones below do not (inspired by "the line" on Google Summer of Code's mentorship application). I originally thought of having two lists side by side, allowing people to move elements from one list to the other, but this was too wide for some screens once the possibility of options selection was added. 

I've also gotten a plain white Google Map (to replace traditional geographic maps) working, as well as overlays. Google maps have a wrapper for XmlHttpRequest called GXmlHttp that should make refreshing the chart based on user-sorted options possible. Now the question is how to efficiently recover information from the database to draw the tree, highlighting which branches lead to software+methods, just methods, or nothing.

Viewing missing methods/software

The key interesting thing about TreeTapper for me is the ability to find missing methods or software. Any list of software and methods will tell you what's available (and isn't trivial to make), but for developers, finding what doesn't exist yet is key. At first, I was just doing a typical treeview (not in the phylogeny sense, but in the nested series of folders sense):

I started adding the beginnings of  bar plots (the red squares above) to show the number of techniques/question available for each topic. The problem with this is that it's very hard to get a quick overview of what's missing: a user has to drill down into each section and remember what's there (sensible display of some of this in with bar plots might help, but it's still not great).  But thinking about it, what would be good to show is an actual tree: for a given starting point (such as a topic: speciation rate), and then all possible descendants (such as all possible questions for this topic). Those descendants available in methods/software get one branch color (say, black), those not get another branch color (gray) [though it might be good to distinguish those present in methods but not software]. Here's a hand-drawn example for the basic idea:

And with colors and labels:

Under this approach, it's easy to distinguish areas with methods/software available (black/solid) from those lacking methods or software (gray/translucent). In the example above, the central dot represents a topic, the first circle represents questions, the second circle criteria, the third perhaps character type, etc. Derrick Zwickl had the good suggestion to allow users to set the order in which options are plotted; I'd also like to allow users to fix certain values (only look for missing methods under a likelihood criterion, for example).

The problem is that this is just a dummy layout, drawn in Apple Keynote, not actually a working image. The question becomes how to make it. I'm thinking of first having a YUI table with the various options to plot (criterion, method, character types, etc.), and then having a second table (or allowing ranking on the first table) where users can drag the options to plot them on the tree in the given order (see an example of something grossly similar here). One problem may be writing the logic to be able to look at all the options for variable Y when it descends from variable X, including which ones are and are not available, when what X and Y are is up to the user (perhaps comparing a cross-join and a left-join postgres table, or something like that, would be the key). Another question is how to actually generate the plot. There are  various Java libraries for interactive data plotting (the first thing I would try if I went this route would be generating an interface with Processing), but many of them failed when I tried them with the most recent version of Mac's Safari, and Java online (and on the desktop, too) always feels a bit clunky to me. There are various ways to make plots on the web (such as Google Charts and Yahoo Charts), but they only have a few mouseover options. I'm actually thinking of using the Google Maps API for this. Using that, plotting points, lines, and areas is now possible, and users can get information on nodes by clicking on them and one can add various javascript functions called onmouseover, onclick, etc. Users will be able to zoom in on parts of a tree. Finally, one can add custom map tiles to replace the Google tiles; in my case, I'd just have a white background, and do all the plotting with polylines and the like. This sort of use of Google Maps been done before; I remember Katy Böhner mentioning this in a talk (though I couldn't find anything on her lab's website), and there are other examples online. 

Well, we'll see how it goes. As with all posts, please feel free to make suggestions in the comments.