Ask HN: What roles have you had the most difficult time filling?

[ropman76]

I am not personally looking for work but I was having a conversation with a friend and he was talking about a particular position he was having a hard time filling due to a lack of local people who have experience in that skill set. So I am curious, what roles are you having a hard time filling on your team?

[chollida1]

People who know how to clean and work with large disparate datasets.

I guess its not surprising given how Jim Simons of Renaissance Technologies fame indicated that of his first 10 employee's almost half were cleaning data.

It's alot more difficult than people realize and you know right away that someone has no real idea of the scale and difficulty of the problem when they suggest that a shell script can solve most of the data issues.

I think Renaissance Technologies actually illustrates just how much a good data cleaning and back testing platform is a real competitive advantage.

A couple of former RenTech people left for Millennium partners and for a couple of years.

Even though these employees were good enough to work at RenTech and had insights into the strategies employed there, they weren't able to be successful on their own without the huge backtesting and data cleaning framework at RenTech.

[lastofus]

What are some if the types of problems that needed to be solved when cleaning data that required heavy tooling?

[itronitron]

Lots of data typically means streams of data, which means processes running 24/7 moving data and files around. Streams, connectivity, and processes can cut out periodically which means you need some logic to reconcile and fill the gap and also restart the processes. You will also need some data QA as it is perfectly reasonable to get 'extra' data, either as duplicates or metadata bleeding into content.

If your data is from disparate sources then you may need to normalize timestamps across records from different sources, you may be dealing with different languages, identical tokens that mean different things depending on the source, different formatting of numeric fields, etc.

This is an incomplete list, the GP probably has a more exhaustive list of problem types...

[pcmaffey]

ie a subset of Data Engineering role

[souprock]

Low-level hacker roles are hard to fill. For example, my posting on the "Ask HN: Who is hiring? (January 2018)" thread is this:

https://news.ycombinator.com/item?id=16057016

We get a little bit of luck with people who started hacking back when it was normal to care about assembly language. For example, somebody on our team used to write cartridge-based games for the Atari 800XL computer, which was an 8-bit home computer system. There are not a lot of people like that though, and many of them don't want to move because they have settled down with houses and family.

We get a little luck with people fresh out of high-end engineering schools like Carnegie Mellon and Georgia Tech. Those schools still prepare students for dealing with low-level hacking.

I think much of the trouble is that many people entering college have a focus on games, web sites, and phone apps. Writing low-level code (hypervisor, emulator, exploits, boot loader, OS kernel, compiler back end...) isn't something that seriously enters the mind of the typical student. People pass (or avoid?) their "Computer Architecture CS351" course with MIPS code, and their "Operating Systems CS302" course with Minix, and then they forget that stuff as fast as they can.

[stevekemp]

Your advert sounds pretty awesome, I used to do a lot of low-level coding back in the day (first z80, then intel), and now I'm rediscovering my love after starting to develop hardware-based projects with Arduinos and ESP8266 devices.

One thing that jumps out is that I know many of the tools you list, but I'm basically AVR + Intel these days, so seeing all the architectures listed is a bit overwhelming.

When I see posts like that though I always imagine it comes down to location and salary. For the right price many people would move, if they're not local.

I've seen too many adverts where people want the kind of skills you'd learn over 20+ years of industry employment, with a salary a teenager could live off, and its not too hard to understand why the same jobs get posted year-in, year-out. (Not that I'm accusing you of that, but it's a slippery slope, and filling "impossible" jobs gets easier every time you increment the salary.)

[souprock]

The more of those things you can deal with, the better. We certainly don't require a person to know about them all. We're trying to redundantly cover all those skills with a few hundred people.

Hmmm. Maybe I'm scaring people off.

For example, we'd hire somebody who is fully focused on x86 if they are good at that. The sort of level we'd be looking for is a person who can recognize the common string.h functions in bare x86 assembly code.

The Arduino usually uses a ATmega8 CPU. I just encountered that CPU, not in an Arduino, and might soon be dealing with it. Adaptability is really desirable; this is a CPU that I've never dealt with before and I'm not about to wimp out.

The ESP8266 has a core based on the Tensilica Xtensa, which I've dealt with.

I think I heard somebody around the office dealing with a Z80, but I'm not sure if I remember that right. Chances are, we've done Z80 work.

It's kind of fun to encounter a new CPU. It's especially neat to encounter one for which step 1 is to write a disassembler.

I certainly do post the same jobs year-in, year-out. That doesn't mean we got nobody. We need more than one person.

Lots of people really won't move. Maybe for a $million they would... but only "maybe"! It is particularly hard to convince people on the west coast that there is civilization elsewhere. There is a fear of being stuck if the job doesn't work out, yet here I am in a city with at least half a dozen large defense contractors and a whole bunch of cyberwar-related startup companies and even some space program work.

[stevekemp]

I suspect you might find more applicants if you focused on family - e.g. look for an x86 hacker, then a powerPC person, then a microcontroller person.

I'm not sure about American, but the Z80 was very very popular in the UK and Europe, being at the heart of the ZX Spectrum and other 8-bit home computers in the eighties (which is when I first got started, hacking games for infinite lives, and removing protection systems).

But either way good luck, it sounds like you're doing fascinating things though your niche is going to struggle it shouldn't be impossible. I know local people who comparable levels of low-level work A/V companies, and malware analysis.

[ioddly]

As a web dev who writes compilers in his spare time for fun, this is fascinating to me. I'd love to have access to more low level work, but I don't know how to find it or "make the leap." It seems like a need that's under-served for both employers and job-seekers.