Hacker News new | ask | show | jobs
by iDemonix 3134 days ago
Looks good. What are you using to get the jobs in the first place?

I recently wrote a scraper in Python using BeautifulSoup that scrapes TechnoJobs, Indeed and some others. It looks for keywords, does some other stuff, then sends the weekly results to my Slack channel.

3 comments

This! There's hardly any uniqueness in listing because everyone is scrapping from the same source.

Neat product but I just wish job boards seperate from the herd by listing unique opportunities that arent mirrored on other sites

You definitely make a good point. On the one hand, everyone who goes to look for jobs would love to use one site and feel confident that they're seeing everything available. On the other, it creates a lot of repetitive listings.

I'm planning to add more value with additional resources for prospective PMs and ultimately I'd like to have unique listings as well. I think aggregating other sites will help build the audience at first though. Otherwise, no companies would bother to share unique listings on the site.

Curious to hear your thoughts though

Uh, what? That sounds horrible. Why would you want to be forced to check n job boards, filtering through hundreds of overlaps, to find the m << n unique opportunities?
Nice! The 3 main ways of getting jobs will be through APIs, RSS, and scraping, so I started out connecting to each type. Indeed and GoRemote API, Mind the Product's RSS, and scraping a few sites using Ruby & Nokogiri, which isn't actually in production yet.

Did you do the keyword filtering when you connected to the services (like pass a query to indeed) or parse the results after?

One of the challenges with Indeed is that there are so many results and it's hard to identify actual product jobs among the results. For example, Microsoft called their PM position "Program Manager", but if you search for program manager you get all kinds of other results that have nothing to do with tech or startups.

Which "other stuff"?

Sounds like a cool thing you build. I would like to use this. Please ping me on my email address - it is in my HN-profile. (You haven't put your email in your profile.)

It doesn't do a lot, and it's far from production ready! Some of the other stuff is just things like filtering out bogus salary ranges or doing the opposite to keyword finding and ignoring jobs that use lots of tech I don't know about. It's quite buggy and hacked together, I tweak it often.

I was thinking of using it to power a small UI like OPs, if I do I'll drop you an email. It's fairly easy using BeautifulSoup though, and was quite fun to learn! I'd encourage taking a look if you're a Pythonist.