Merge codebase

When I left NESCent, I deposited TreeTapper's code in Google Code ( http://code.google.com/p/treetapper/ ). Subsequent development happened on a privately-hosted version of that code (I was worried about accidentally committing a file with database access passwords). I've since learned a lot more about SVN and have svn:ignored the dangerous files, as well as files that aren't really things to put in version control, such as cached images. Now, changes made on the Google Code repository will be made live on the site. This should make it easier to collaborate with others going forward.

Viewing missing methods/software, part VII: PHP+GD

I was getting dissatisfied with Processing for visualizing missing methods. It was nice to have animation of the diagrams, but it was just too brittle: it would only work in certain browsers, it stopped working sometimes when you moved between tabs or windows, it was hard to update and keep working. This is probably more due to my lack of experience in the language and the odd way I was using it (read a variable stored in Javascript on a page to talk to a PHP script to talk to a Postgres database and display the results in real time) than any inherent flaws in it. I had thought about going to Flash, but that also involved learning a new programming language. I had recently started using PHP with GD to visualize progress of my jobs on Duke's computing cluster and found it easy to use (though I'm having issues with transparency). I decided to try this with the method visualization (I wanted to get TreeTapper cleaned up a bit more before a job interview next week). I wrote a script that reads information stored in GET variables and saves an image and image map. In front of this I wrote a controller script that only calls the drawing script when an entry in the database is newer than the saved image and image map (using ideas from my Google Summer of Code student Paul McMillan, who put sophisticated caching in his DBGraphNav). The image and image map are displayed in an iframe in the calling page. There's a Javascript function that calls the controller script as a user chooses which options to display. There's also a Javascript function that uses a Panel object from YUI to display a lot of information about each node on the diagram on mouseover in a floating panel. This is a lot more flexible than the Processing implementation. For example, you can now go to a summary page for each node or click on an entry in the floating panel to go to an info page on just that item (such as a program).  Caching the images makes display a lot faster, though only a subset of possible combinations can be cached. The cache of images also provides some nice pictures for screen savers or talks.

You can try out the new interface here.

Here's a picture of the new display:

And some of the cached images:

Better input

I've been adding methods and software to the DB slowly, I think due to a somewhat awkward interface. TreeTapper is currently configured so that various traits of a method or program (criterion, character type, etc.) are selected from pull down menus or tables. If an element isn't there (such as a particular format of tree file), I currently have to go to a separate page, add the item, and then reload the first page. I'd been playing a bit with having popup panels for this entry and have decided to just do them. This will simplify, for a user, adding things to a database: she or he can just go to the add method, add software, or add reference pages.

Flash rather than Processing?

I'm now wondering whether Processing was a bad choice for visualizing missing methods (though it's still better than the horribly slow Google Maps implementation). It results in Java applets, which can take quite some time to load on a page (see other general criticisms here). I had to do some hacky things (use Java code rather than stuff built into Processing) to get the applet to talk to Javascript on the page to find what items a user wants to examine, which might not be stable as Processing develops [and it probably prevents me from just converting to Processing.js or the like]. Also, on the stable site (http://www.treetapper.org, rather than http://treetapper.nescent.org), the existing Java applet I made doesn't work (to see a working version, go here). I've tried playing with the code (changing it to point to the stable site, of course, and looking at other potential issues) and it still doesn't work. Flash animations seem to work much faster and more stably across many browsers. Some versions (>4.0.5, <5.3.0) create Flash animations (the code's been moved into a separate install for later versions of PHP), so I could use that. I could just code it in ActionScript and somehow compile it. Either way, I don't know anything about coding Flash animations, how they can connect to the database and page elements, how to write them without depending on expensive software, etc., so it'd mean a bit of work to learn a new language. Any ideas?

Viewing missing methods/software, part V: Processing

In my previous post, I was  bemoaning the slow speed of the visualization chart using Google Maps. Looking around in the Google Maps group, it seemed that there wasn't a good way to speed up the drawing of many markers, except for drawing them in another program first and storing them on image tiles. Since I was going to have to draw markers in a different program anyway, I decided to try using such a program by itself without involving Google Maps. First, I thought to create a static image using GD with PHP and then do an image map on top of that with javascript, but GD wasn't working on the NESCent server (this was fixed in just a couple of hours, as is common with NESCent's great tech support). While it was being fixed, I decided to learn Processing — I've been impressed by the quality of some of the diagrams it creates, and having a dynamic diagram might be more useful for users. It has good documentation and can parse XML, so within a few hours, I learned enough to be able to create the diagram below. 

Following advice from the TreeTapper design consultant (my wife), I made the nodes smaller and changed the color slightly, but otherwise it is the same as the Google diagram (though I don't have popups or mouseovers working yet), and is much faster to create. It does require users to have Java installed in their browsers, but might allow cool features later on.

Viewing missing methods/software, part IV: Sort of working

Well, I sort of have the visualization working. Users can choose which elements to graph (on Firefox) by dragging boxes to move them above or below a plotting line, and they can choose to limit to plotting only one of many options. See below.

On clicking the  update chart button, a tree is plotted using the Google Maps API, with branches colored based on whether they have no methods or software, methods but not software, or both methods and software [I originally had it update the map automatically on any change, but this is too slow]. Users can click on nodes to get an info window showing the choices made going from the tree root to the tips and links to any relevant software or methods. Just mousing over a node tells a user what option that node represents (i.e., "Treetype: Unrooted, polytomies, incomplete tree"). If a user has chosen just one setting for an option (as for criterion, below), the edge leading to that node is shorter and light gray edges are shown to indicate the options not examined.  See below (click to make larger).

The end product is beautiful and information-rich. It's also VERY slow. It takes literally minutes to generate the plot, and then >10 seconds between clicking on a node and getting the info window. Zooming in or out or moving the plot also take an agonizingly slow time. Getting info from the script that talks to the database to make code for the map takes a while, about 30 seconds, but the real slowdown comes when drawing the map. Google maps are slow with many markers and polylines: the map above has 221 markers and even more polylines, with the three circle polylines having 360 points each (using fewer makes a plot that's too rough). I'll have to decide what to do. I'd stayed away from Processing due to usability concerns, but it seems Google Maps isn't so great, either. I've read a little about generating image tiles rather than markers to speed up the map -- I'll look into this and other options. 

Citation counts

When deciding which method to use, or on what areas to focus development, the popularity of related references matters. For example, if method A is used 20 times more frequently to answer a given question than method B, in the absence of other information, a naive user should probably use method A. Just counting total citations could be misleading, though: a good new method only available in the last year will take some time to gain as many citations as a poorer older method, despite acquiring citations at a faster rate. Thus, both number and rate matter. I'm thinking of showing both the total number of citations and rate of gain of citations over a year or so; another way of displaying related info would be comparing the number of citations for a paper with the median number of citations for papers published in the same year (perhaps limiting the papers in the reference set to those similar in scope to the paper of interest). To get this info, I'll need citation info, which is not generally available (see earlier post). I'm using the number of hits in Yahoo, using its search API (which gives slightly different numbers than its html form search results) for both all pages and only PDF-formatted pages with an article's title phrase, last name of first author, and publication year. The tricky things getting this to work were converting apostrophes to html characters and making sure to include the title as a phrase, rather than as a string of words. This approach has a few disadvantages: while web hits probably correlate with how many times a paper is cited (early work suggests this is roughly true), it is not wonderfully correlated (but it can pick up hits for interesting new papers faster than waiting for later citing papers to appear), plus it is easier to mislead (I could add my papers' titles, authors, and years as signatures to all my posts and then start posting on various forums: TreeTapper would see my papers as very popular). Perhaps with the upcoming release of the new CiteSeerX, I can use that system to track citations better. I've set up TreeTapper to store the number of hits for each paper every two weeks; this will allow me to track how the popularity of articles changes through time to recover rate rather than just number of citations. As this recording is new, I'm now only showing total hits until the data are recorded over more time intervals.

Progress so far

TreeTapper is hosted by NESCent and uses a PostgreSQL database as a backend, an assortment of PHP scripts for generating pages, and the Yahoo User Interface Library (YUI) for a front end. Reasons for these decisions:

• NESCent hosting: Free, stable, well-supported, appropriate for the project, and allows things like use of mod_rewrite to make creating RESTful site easier.
  
• Postgres: NESCent supports this rather than MySQL; triggers and views will be useful (though now also allowed in MySQL). I had thought about using arrays for some fields (a Postgres-only feature), such as when a particular bit of software can read multiple tree formats, but will instead use additional tables, following recommendations from NESCent's IT staff.
• PHP scripts: I had looked into using Ruby on Rails, CakePHP, or other frameworks for development, but there seemed to be a lot of overhead in learning them for the benefit I'd receive.
  
• YUI: This is one of the many libraries for Ajax development. YUI is feature-rich and has both great documentation (with many examples) and an active user forum, both important for me.

I've also created the database schema and a way to securely log in (using PHP sessions). Next, I'm adding the ability to store and look up references in Connotea (using a Perl API) and other reference-parsing tools so that I can start adding papers to the database; I'll also create forms for adding methods and software and start actually adding information.