[davidmr]

That's very true. We do actually do that with our production team. It's split into "ops", which is the day-to-day work and "engineering", which is the more complicated, more interesting work.

We hire junior people we think are smart into the ops roles and train them to eventually shift over into engineering. Some people can do it in a year or less, some a lot longer, and some just don't work out at all (or actually like doing the more routine work). Those people are definitely easier to find. We just talk to a lot of people and pick the ones we think are the smartest and will mesh the best and train them up on the rest.

In some cases, though, you just really need to get experienced people and can't wait the amount of time it would take to train.

[vonmoltke]

What kind of experience are you looking for? Is it possible you are being too selective or misjudging the ability of experienced applicants?

[davidmr]

That was definitely part of it, but we've changed our strategy a bit in an attempt to compensate. At first we were looking for people who had a lot of HPC-specific Linux experience. This, however, is a relatively small field because the cost-to-entry barriers are enormous if you're looking to do it at any great scale. That definitely restricts the set of potential candidates to a small subset of the set of "Linux people" (people from gov't labs, research universities, very large corporations, etc.)

Since we're willing to pay relocation expenses and buy people out of non-competes, we were doing a nation-wide search, but as a generalization, university and gov't employees are usually there because they like the relaxed environment and are willing to sacrifice some pay for it. (I have worked at both, so that generalization is drawn from only my experiences and those of my friends.)

So after getting few bites there, we rewrote the job description to just be very good with Linux in general, hoping to get a really smart person with skills that complement ours and then being able to train them in the HPC-specific stuff. That increased the number of applicants, but also dramatically decreased the quality-applicant to applicant ratio.

You're definitely right that I could be misjudging the ability of some of the applicants, but a great majority can't pass the Linux equivalent of FizzBuzz, even though they list 10+ years of experience.

[ciupicri]

What would be the Linux equivalent of FizzBuzz?

[davidmr]

I'm thinking of it as a test for the simplest possible thing I can ask someone that will tell me immediately if they're worth talking to or not. I mentioned this in another post, the one I use that people bomb most often is: "I want to run this command on 100 files. Write me a for-loop in the shell of your choice to do it."

[yourapostasy]

That's kind of terrifying that someone can't do that can say they are an SA. Did anyone you interviewed mention xargs as a BTW solution after presenting the for-do loop? I'm currently trying to figure out if one of the SA's at a current client actually does not know how to use tar, gzip, and scp, so I know where you are coming from, but still.

When you previously said "complicated applications in (python|perl|ruby)", did you mean someone who knew both front- and back-end? That implies someone who also knows Javascript, CSS, HTML, and a few databases, not to speak of one or more web frameworks, proxies, load balancers, etc.

When you say "debug and port C/C++ programs", did you mean someone who knows enough about Windows to port a program written there to run on Linux? Or someone who knows enough about BSD to port to Linux? Or someone who knows enough about [AIX|HP-UX|Solaris] to port to Linux? Port what, pure userland programs or stuff that messes around with ioctl calls? Debug from a hand-built in-memory circular log, core with no symbol table, using Dtrace, or just some judicious printfs or logging?

Depending upon how you specify what you want, you don't have to go far these days to stray a long distance from a "Linux SA" role, where even a "Senior Linux SA" job title won't cut it; you are moving closer to a Free Electron, and those people don't come cheap.