Hacker News new | ask | show | jobs
by shavenwarthog2 4443 days ago
Agreed. I did a multipart Yahoo Pipes project to find my current apartment. It grabbed info from two sites, tossed out the uninteresting ones, filtered it a bit, then texted me if a new apartment in my price/location range appeared.

Very useful, if a little awkward. The Huginn project sounds like a great alternative!
1 comments
scuba7183 4443 days ago
Just curious, how many hours of work did that take? Do you think it was worth it?
link
shavenwarthog2 4443 days ago
It's hard to tell -- I tweaked it quite a bit, and rewrote it from scratch 2-3x. I'd say 4-10 hours.

Was it worth it? As a programmer, no. I'm very familiar with scraping (raw) web/RSS feeds for data, then processing it. I was hoping Pipes would have enough intelligence, so that I could subscribe to (cooked) data sources, then split and refine the results.

In practice, Pipes worked, but the data always required further post-processing, which was awkward to do in Pipes. You have to be a dev to understand what your system is doing, but you don't have easy access to all the standard dev things.

I look forward to seeing Pipes take off, or another technology (Huginn? Ifttt?) replace it. It was a lot of fun to wire things up graphically then for example get a text when someone's RSS feed changed.

link
hyp0 4443 days ago 

  You have to be a dev to understand what your system is doing, but you don't have
  easy access to all the standard dev things.
Interesting, this mismatch may be a good description of the problem of visual languages.

Curious: what do you think is the minimal subset of unix tools to do this? i.e. instead of pretending the problem is simpler than it is, accept the complexity, but minimize it.

I'm thinking of a tool like "jq" (sed for json) for json data sources... but I don't think its raw-text manipulation is up to the task (and of course you need tools to monitor the feeds etc).

link
TylerE 4443 days ago
Trying to manipulate structured data as text makes about as much sense as parsing XML with a regex.

http://stackoverflow.com/questions/1732348/regex-match-open-...

link
shavenwarthog2 4442 days ago
Python :-) there are libraries specifically for parsing malformed html. I'm happy using Unix tools for scraping and parsing, but you run into a brick wall rather quickly. Python is more reliable, flexible, and easier to integrate.
link
shavenwarthog2 4443 days ago
I published a few pipes -- enjoy! http://pipes.yahoo.com/pipes/person.info?guid=W4YBIUCXEVLHMS...

They're all quite simple. The most complex one uses the "parse location into lat/long" Pipes feature to automatically find me jobs in the Los Angeles area.

link