That's really cute, but since you've solved the maze, can you tell me what the copyright encumbrance on that code is? Is it open? (By which I mean, is it worth my while to write my own crawler?)
I'm going to wrap it in a Perl module so the whole thing comes right down from CPAN, just like SQLite. This is exactly what I've been waiting for, for years.
Scheme 9 from Empty Space -- A Comprehensible Scheme Interpreter By Nils M Holm, 2007-2010
* Do whatever you want with this code. * If the code doesn't work, don't blame me.