[_urga]

I was thinking through something like this just last night, after reading up on Smalltalk and system images. Are you loading each domain in an iframe? How did you manage to overcome Twitter's clickjacking protection? Using the new sandboxed iframe attribute? High five!

[CrabDude]

It's done via proxy, node.js, and websockets =) No iFraming. Since we're proxying, we can dynamically hijack the links and insert JS.

[jbr]

What are you using for server side dom manipulation? jsdom? apricot? node-xml? libxmljs? I spent a lot of the weekend working on a webcrawler, but couldn't find a xml parser that didn't choke on the internet-at-large.

Any chance you'd consider open-sourcing?

[lftl]

I haven't seen their code, but I'm not sure why you'd need server side DOM manipulation for this. I'd implement all of that in the browser, and just let the server handle passing events back and forth.

[jbr]

Since they're proxying the page, they replace all <a>s with proxy <a>s, as well as adding a <script> and a <div> with some content to the bottom. Try viewing source on http://starcraft2destroyedmymarrage.no.de:3000/app/4

[defen]

Do you have any examples of the kinds of things that libxmljs choked on?

[jbr]

I don't recall the urls - I was recursively crawling from a user-provided seed url and had trouble with apricot, node-xml, and libxmljs. I had the least time to play w/ libxmljs because of hassle w/ joyent --- I had to compile v8 and node to link against and scons wasn't playing nice w/ the build environment.

--

Edit: I should add that I'm using libxmljs in production (http://newsbasis.com/news) as an rss parser (streaming sax-push-parser) and it works quite well for that sort of xml.