[zenspider]

perl threading was (and I'm sure still is) absolutely horrid.

I'd be interested in seeing the results of a rewrite not to C, but to python or ruby where the threading support is much much better. Then you could rewrite functions at a time in C as needed, but not have the extra burden of rewriting the whole thing.

I totally agree with the rest of the approach. Going low tech and using unix tools is a very good way to reduce overhead, increase parallelism, and delay calculations. One of the nice things about this approach is you can cobble up another $50 unix box to do some of the bulk processing via nfs or other means.

Congrats... It sounds like a very interesting project.

[kaens]

Threading support is better in python than in perl? I'm not familiar with perls threading model (or libraries), but this seems wrong - considering things like the GIL in python.

Could you elaborate a bit on this?

[zenspider]

I'm much more familiar with threads in ruby. It would actually be a real boon to this project imo.

I don't have as much experience with python threads as I do ruby. I've toyed with the internals of python and perl as well and know perl to be a wasteland and python to be rather elegantly clean. So I would expect python's threads to be better implemented. Poking at it, it does appear to be so.

[jmtulloss]

Doesn't ruby only have green threads?

[michaelneale]

Yes - unless its jruby then they are non green (forget the term for it at this time of the morning).

[zenspider]

yes... and they're great. Esp for situations like the OP was describing. Fire off a ton of incredibly lightweight threads and they'll quietly sit waiting for IO... perfect.

[kingkongrevenge]

The thing with perl threads is that pretty much nobody uses them. They work fine, as far as I know, but with a default 16MB stack size and "share nothing" semantics don't start more than a few. Perl is more unix centric and people just prefer to fork and use the well baked IPC mechanisms. Various libraries such as POE make fork+IPC easy enough that it's hard to see the need for threads in the kinds of domains where a language like perl is applicable.

[kaens]

people just prefer to fork and use the well baked IPC mechanisms

Ahh, that makes sense. I'm willing to bet that the vast majority of perl code runs on *nix systems (I know that with the exception of one app, all the perl I've ever written was only run on linux servers).

[zenspider]

they certainly didn't work fine when I tried to use them... after they'd gone into production releases of perl, the canonical demo/example scripts that were out there all crashed the perl runtime in a fiery horrible death.

I can only hope that it has improved since then, but I've moved on to greener pastures.

[kingkongrevenge]

They were clearly marked as experimental in the documentation all the way up to recent 5.8 releases. From what I hear they work fine now.