[michaelochurch]

You nailed it. The work quality you get if you're in the R&D world you discussed is quite high. It's like being a data scientist at a startup, but with more money and an environment that is less political.

You have to convince people you're more than "just a programmer" to get good work. A quant, a crack low-latency guy, a data scientist, possibly an "architect". By age 30, if you're not some X (as in: "X Who Programs") along with being a good programmer, you start losing.

http://michaelochurch.wordpress.com/2012/08/26/xwp-vs-jap/

[cmccabe]

By age 30, if you're not some X (as in: "X Who Programs") along with being a good programmer, you start losing.

Or, you could work in an industry where software engineers are respected and valued because what they do is part of the core business. If you work at a company whose core business is X and you don't do X, then of course you are going to feel peripheral-- because you are. In computational finance, if you're not a quant, then you're just a member of the support staff and should expect to be treated as such. I fully believe that those firms would contract out this work if they could, but because of confidentiality reasons, they can't.

[michaelochurch]

Right. To be fair to finance, there are a lot of companies (and groups within large institutions) that do value technology. Arbitrageurs, for example, rely on technology. (Financial risk in arbitrage is extremely low; but the danger is that, if your technology sucks, you'll be running at a high burn rate and making few trades.) People are starting to get it, even if it takes a long time.

The across-the-board startups = good / finance = bad mentality makes no sense to me. It may be true in the aggregate, but I haven't seen enough data and what I have seen shows no correlation. If you're in the back office of a huge bank, you're probably not in the best environment for an engineer. On the other hand, high-frequency trading requires a certain technical acumen that's fairly rare and that people will pay a lot for... and if you're really good at that stuff, a lot of the politics works itself out.

I think it's essential to become an XWP ("X Who Programs") even in mainstream software shops. Unless you have a national reputation as a great software engineer, you need some credibility that's independent of your performance as an engineer. Why? Because if the architecture's bad or nonsensical, good engineers won't perform well because they don't think in FactoryFactories and inheritance hierarchies. You need control of the architectural landscape in addition to engineering ability to perform at a 1.5+ level. Otherwise, if you take a typical software job, you're at the risk that you perform at a mediocre level because of architectural decisions made by other (less competent, but more powerful) people and never be able to convince people that you can take the architectural reins. You need that X (data science, project management, quant finance) to convince people that you're substantial regardless of your performance in the particular environment that's given to you.

I am doubling down on data science, machine learning, and statistics right now. (I was a math major in college, and had a couple of 50+ Putnam scores, but I've been away from math for too long and have to review linear algebra to make sense of a grad-level textbook these days.) What I learned in my last startup (where the unambiguously best engineer was demoted and an overpromising mid-20s bullshitter became unofficial VP/Eng, and threw the company into a "rearchitecture" that nearly killed it and ruined the culture) is that architectural issues get political fast. Very fast. Like, tachyon fast. To perform well as an engineer, you need to influence (and, one hopes, to improve) the architectural landscape, and you need some "hook" that is independent of said landscape in order to convince people that you're smart enough to change it as you need.

[cmccabe]

I think it's essential to become an XWP ("X Who Programs") even in mainstream software shops.

Well, it wasn't essential to Linus Torvalds, or Doug Cutting, or Joel Sposky.

If you enjoy data science and machine learning, do data science and machine learning. There's a lot of cool stuff going on in that field. But don't use it as an excuse to look down on people in different fields. There's too much of that going on already (and yes, developers do it sometimes too.)

[dagw]

I can't speak for Doug Cutting, but is my understanding that both Torvalds and Spolsky are XWP for values of X=Really good at managing projects and programmers, and in the case of Spolsky also pretty good at the whole running a business thing.

Certainly in Spolsky's case I don't think he writes much if any code at Fog Creek or Stack Exchange. Torvalds probably still codes a fair amount, but certainly when it comes to Linux most of his value comes from the architecture and management side of things rather than the lines of code he writes.

[cmccabe]

Everyone is an "X who Ys" for some value of X and Y. The question is what's your main job? If the answer is writing, reviewing, designing, or philosophizing about code, then your main job is software engineer. All three of the folks I mentioned fall in that category.

I had a friend in college who was extremely proud of his double major in biology and computer science. He went on to get a PhD in bioinformatics. He was fond of telling me that being "just a software engineer" would never be as exciting as the work he was doing in his specialized field (not going to go into details, but it was related to pharmaceuticals.)

A few years later, the pharmaceutical industry is in a slump and he can't find a job in his specialized field. So he's working on web design for a while, hacking PHP etc. I'm doing the same thing I've always been doing, working on distributed systems, and it's been very rewarding. (Before you ask, I did try to find a job for him, but he didn't want to move to the west coast.)

The moral of the story is to do what you really like and are good at, and don't worry about people telling you that you absolutely MUST shift your focus to X to succeed (where X = being a manager, being a quant, learning biology, etc).

[michaelochurch]

But don't use it as an excuse to look down on people in different fields.

Oh, I don't look down on "pure" developers (as opposed to XWP) at all. I just think that it's hard to establish a reliable stream of good work on engineering cred alone, because the people who make decisions in most businesses can't tell the real deal from the charlatans in software engineering. The X is the "hook" that gets you enough clout to ensure you get interesting work and continue to get better as an engineer.