Open Sourcing EVE Crawler
Multithreaded JavaScript has been published with O'Reilly!This is a project I started working on and abandoned in 2009. It is a spider which was specifically built for crawling websites which contained EVE Kill Mail's. In the game of EVE, every time you kill a player, an in-game ‘mail' is sent to you containing information. Players would copy this informaiton and paste it into a website, which would show statistics about the kill.
The purpose of this project was to take information about all of the kills and display them on an interactive 2D heatmap. There would also be a slider which represented time, and a user could slide this slider. The purpose of the project was to show which areas in space were the most dangerous.
Unfortunately, I stopped playing EVE and abandoned the project. The crawler, which worked in 2009, is likely no longer useful. It would download pages and run regular expressions to find data, as most ‘kill mail' websites used the same software. But, I'm sure the software has changed by now.
I honestly don't remember what the state of the project is. I do know that the crawler works just fine; I was able to grab a couple million records. The flash map stuff is probably all broken and whatnot. Back in 2009 we didn't have any fancy canvas rendering, a better developer would replace the flash with canvas.
The crawler wasn't as efficient as it could be. Most of the slowness would be due to network latency, so paralell downloads should be used.
None of the interface development for clicking regions, doing heat maps, or scrolling through time was ever implemented.
Released under the BSD license.
Thomas has contributed to dozens of enterprise Node.js services and has worked for a company dedicated to securing Node.js. He has spoken at several conferences on Node.js and JavaScript and is an O'Reilly published author.
