Viewing missing methods/software, part VII: PHP+GD

I was getting dissatisfied with Processing for visualizing missing methods. It was nice to have animation of the diagrams, but it was just too brittle: it would only work in certain browsers, it stopped working sometimes when you moved between tabs or windows, it was hard to update and keep working. This is probably more due to my lack of experience in the language and the odd way I was using it (read a variable stored in Javascript on a page to talk to a PHP script to talk to a Postgres database and display the results in real time) than any inherent flaws in it. I had thought about going to Flash, but that also involved learning a new programming language. I had recently started using PHP with GD to visualize progress of my jobs on Duke's computing cluster and found it easy to use (though I'm having issues with transparency). I decided to try this with the method visualization (I wanted to get TreeTapper cleaned up a bit more before a job interview next week). I wrote a script that reads information stored in GET variables and saves an image and image map. In front of this I wrote a controller script that only calls the drawing script when an entry in the database is newer than the saved image and image map (using ideas from my Google Summer of Code student Paul McMillan, who put sophisticated caching in his DBGraphNav). The image and image map are displayed in an iframe in the calling page. There's a Javascript function that calls the controller script as a user chooses which options to display. There's also a Javascript function that uses a Panel object from YUI to display a lot of information about each node on the diagram on mouseover in a floating panel. This is a lot more flexible than the Processing implementation. For example, you can now go to a summary page for each node or click on an entry in the floating panel to go to an info page on just that item (such as a program).  Caching the images makes display a lot faster, though only a subset of possible combinations can be cached. The cache of images also provides some nice pictures for screen savers or talks.

You can try out the new interface here.

Here's a picture of the new display:

And some of the cached images:

Viewing missing methods/software, part VI: Processing done!

I've written the code to generate the missing methods/software tree diagram using Processing. The relevant page is http://treetapper.nescent.org/findneed.php . I've designed it so that as data streams back from the database via a PHP script, Processing draws this on the diagram in real time. As users update the sortable list of options (tree type, general question, etc.) (using YUI drag and drop), a Javascript function updates the string (stored in a Javascript variable) that is passed to the PHP script. Processing checks this string (so, Java talking to a Javascript object), and if it has changed, Processing closes its old connection to the PHP script and opens a new one using the new options. It thus dynamically updates the chart and is far faster than Google Maps API. It's also easy to do sophisticated animations easily in Processing: currently, I have the nodes flying out of their parents, zooming and shifting the image, and point highlighting on mouseover. These actually don't slow down the rendering: I have the script written so that it only adds a new node to the diagram once the previous node has reached its destination, which sets an upper limit on rendering speed, but I override this and put many nodes on the tree at once if there's a backlog (>10 nodes read from the server but not drawn yet). There is rarely such a backlog, indicating that nodes are being drawn as fast as the server is passing them to Processing. It's pretty cool to be able to visualize where our field needs work (or, rather, where I need to fill in the database) using a dynamic interface, and also surprising that it wasn't too bad to program (especially considering I didn't know Processing/Java, Javascript, PHP, or Postgres when I started this in November 2007). Here is a video showing the new site being used (also available here); you may need to widen your browser window to see it all. As always, suggestions are encouraged.

Viewing missing methods/software, part II

It appears the strategy of using Google Maps with YUI display of items to add to the map will basically work. Rather than doing drag and drop between YUI datatables, I'm using just the drag and drop YUI code on a list of options, each with the possibility of limiting it to one item (for example, one can first organize by optimality criterion, deciding to show all or just likelihood). The interface is based on the YUI example, but with just one list, with one element of a different color so that options placed above this element appear on the tree while ones below do not (inspired by "the line" on Google Summer of Code's mentorship application). I originally thought of having two lists side by side, allowing people to move elements from one list to the other, but this was too wide for some screens once the possibility of options selection was added. 

I've also gotten a plain white Google Map (to replace traditional geographic maps) working, as well as overlays. Google maps have a wrapper for XmlHttpRequest called GXmlHttp that should make refreshing the chart based on user-sorted options possible. Now the question is how to efficiently recover information from the database to draw the tree, highlighting which branches lead to software+methods, just methods, or nothing.

Progress update

I've started adding some methods to the database. It always feels a bit odd to shoehorn a method into a database like this, but it should be useful in the end. I'm also working on a better (i.e., working) findmethod/tool page. As users select options, the table of methods will automatically update. Maybe. I've also updated to YUI 2.5.1.

Upgrading to YUI 2.5.0

I'm upgrading the site to use YUI 2.5.0 (the previous version was 2.4.1). I decided to do this because the new version has many more features in data tables, and much of the utility of the TreeTapper site comes from interaction with data tables. The process of upgrading isn't too bad: the only problem has been needing to either add a "?" at the end of datasource files and then using initialRequest, or at least setting initialRequest to "" rather than the new default of "null" [under 2.4.1,

this.myDataSource = new YAHOO.util.DataSource("templates/vocabtable_js.php? table=applicationkind");

worked, while under 2.5.0, it is transmitted as

[URL]/templates/vocabtable_js.php?table=applicationkindnull

which won't work. The solution is to do either

this.myDataSource = new YAHOO.util.DataSource("templates/vocabtable_js.php?");
[...]
this.myDataTable = new YAHOO.widget.DataTable( "applicationkind", myColumnDefs,this.myDataSource, {initialRequest:"table=applicationkind"});

or

this.myDataSource = new YAHOO.util.DataSource( "templates/vocabtable_js.php?table=applicationkind" );
[...]
this.myDataTable = new YAHOO.widget.DataTable("applicationkind", myColumnDefs,this.myDataSource, {initialRequest:""});

The first option seems better].

The new version also has parsers (perhaps they were there before, but I missed them) that allow text coming over XHR to be converted to numbers, allowing numerical sorting.

The new version has at least a couple of downsides. First, making the tables seems slower (see this discussion), which is a problem when you need big tables, as TreeTapper does, and column headers are now drawn separately, and it can take a long time (>5 seconds) for them to line up with the corresponding column in the table.

Google charts

I've added some Google charts to the front page of the TreeTapper site. I'm using googlechartseasyphpclass v 1.02 to more easily generate the code to call the charts (it involves converting numbers to letters for plotting, for one thing), though it does limit flexibility a bit (but the source code for the PHP script can be modified easily). YUI also has a new charts API, but it requires a very recent version of Flash for people to use (more recent than I had in FireFox) -- Google can simply create a png, which is convenient. The code to make the data to pass to the charts takes a little while to work (dozens of postgres calls): I might just update this daily and have the site call a saved version of the data. The charts I've made allow tracking of how much the database has grown in the previous month as well as breakdowns of references by year.

Progress so far

TreeTapper is hosted by NESCent and uses a PostgreSQL database as a backend, an assortment of PHP scripts for generating pages, and the Yahoo User Interface Library (YUI) for a front end. Reasons for these decisions:

• NESCent hosting: Free, stable, well-supported, appropriate for the project, and allows things like use of mod_rewrite to make creating RESTful site easier.
  
• Postgres: NESCent supports this rather than MySQL; triggers and views will be useful (though now also allowed in MySQL). I had thought about using arrays for some fields (a Postgres-only feature), such as when a particular bit of software can read multiple tree formats, but will instead use additional tables, following recommendations from NESCent's IT staff.
• PHP scripts: I had looked into using Ruby on Rails, CakePHP, or other frameworks for development, but there seemed to be a lot of overhead in learning them for the benefit I'd receive.
  
• YUI: This is one of the many libraries for Ajax development. YUI is feature-rich and has both great documentation (with many examples) and an active user forum, both important for me.

I've also created the database schema and a way to securely log in (using PHP sessions). Next, I'm adding the ability to store and look up references in Connotea (using a Perl API) and other reference-parsing tools so that I can start adding papers to the database; I'll also create forms for adding methods and software and start actually adding information.