[lxglv]

Another thing that I'm really surprised about is the salary range for the C++ developers with RTOS experience. In case we compare this with frontend React developers, then difference will be more than 50% (React guys will be paid more). According to my understanding C++ development for projects that require RTOS is much more demanding in terms of skill and corresponding experience. I would also assume that there are much less developers with VxWorks experience: within my region I may find around 40 people with VxWorks in their CV on the job board against 1000 developers with React experience. I really hardly get the economics of the modern IT job market.

[pingyong]

1. There's a lot more demand for simple web apps/interfaces in areas where there are potentially absolutely insane gains to be had from very simple automation.^1

2. Most devs I know at least consider writing simple web apps/interfaces as the most boring work you could possibly do.

^1 (I've seen 1000+ employee companies with workflows that include multiple people simply sorting and searching through data in hundreds or thousands of files manually, something that any PHP or Javascript script would be able to do in seconds. One time they simply didn't have an interface between two apps and there was someone who's entire job was essentially to type the data from one app into the other. A motivated high schooler would have been able to write something that would have saved 4000-6000 hours of work per year. These kinds of insanely low effort automation opportunities simply don't tend to exist in embedded areas, and it will probably take at least 10 years before most of them are gone from small to medium sized businesses.)

[usrusr]

The market for web developers is heavily distorted by companies that effectively own winner-takes-all markets and by a wide range of investor money bonfires chasing similar dreams (not just the unicorns of Softbank and Vision Fund but also a very long tail of much smaller investors and investments).

This isn't quite as prevalent in hardware applications where even well funded startups have to be concerned about actually achieving positive margins and cannot just hope for growth to eventually outpace whatever cost structure they have burdened themselves with.

[atlgator]

I was an architect for a custom multi-platform RTOS built for many core NUMA systems (C/C++, Ada, Java). Most challenging work I ever did because you can't just look up the answer on Stack Overflow. Took a job in IT for 40% more. Now I'm bored to tears and hate my dumb coworkers who think web development is so hard.

[rkangel]

I live and work in Cambridge UK, and it seems to be the other way around here. Cambridge's history is more of hardware based startups (particularly wireless related ones), so the demand mix is probably quite different. I have always done very well with my electronics/software mixed skill set, and it was a surprise to hear (in a previous HN discussion) that that wasn't as valuable elsewhere.

[aardvark179]

It’s not a simple embedded vs. front end divide. I think Cambridge has generally funded and valued companies that are see as solving, ‘hard problems,’ and, rightly or wrongly that viewpoint favours backend stuff and hardware.

I’m not sure this is a problem though. Many large companies are not hiring in the valley because the high level of compensation is making it uneconomical.

[rhipitr]

In my experience, front-end projects always seem to get more attention and visibility from product teams, and the business in general. This seems to happen even when it’s a nascent product that doesn’t make any money yet, and older, more boring systems responsible for large percentages of company revenue often get ignored. I’m not sure if this is related to pay in any way, but it definitely seems like the visibility of front-end apps, and being able to showcase them to wider audiences, doesn’t hurt.

[jonp888]

I see it like this: In large engineering companies doing embedded systems, software developers are seen as fabricators who produce an item according to a specification, like welders or painters. A web developer in a tech company has a much higher status in the company.

[panpanna]

How many hw startups do you know of?

Sometimes I feel SF is all SW and HW has moved to Shenzhen.

[non-entity]

Yeah I've been thinking about moving to lower level software development or even embedded and a ton of the more interesting sounding jobs seem to be outside of the US

[devonkim]

I'm a former embedded software guy and I can explain a bit more. Hardware is extremely capital intensive, takes a long time for any return, and the margins tend to be pretty mediocre. Around the mid-2000s many embedded developers wound up getting siphoned off into mobile development (back to around Symbian times) which was far higher up the stack than doing anything with micro controllers, RTOS, etc. The embedded software community is mostly a bunch of electrical engineers that learned software second and this can be a small handicap against pure software people that learned hardware far later. Absolutely there are still great engineers doing embedded work in the US, but it's a shrinking market in a lot of ways and very regionally concentrated (not just _any_ metro area will do for stable employment - places in Texas or outside Boston were dominant).

I left embedded dev because I saw this coming and I'm just a slow developer that wouldn't have cut it with the demands of mobile software emphasizing cranking out stuff faster when that's just not how I work.

[pjbk]

And not only that. Even if there is money on the table or lives at stake, many companies choose to outsource embedded development instead of increasing salaries to attract top talent. Just recently Boeing software practices were heavily criticized concerning the 737 Max fiasco [0]. Sadly it has become a tendency not only in the avionics industry.

[0] https://www.reddit.com/r/boeing/comments/c6tknw/boeings_737_...

[exikyut]

Out of curiosity, what do you do now? Did you get out of software entirely? (I'd say I'm kind of slow too.)

I get the impression embedded is still a somewhat slow market. Or has that slowness meant it's dried up really badly?

[devonkim]

I was young and am a mere hardware and server nerd at heart so I wound up down the cloud / DevOps / SRE or whatnot path unfortunately a little too early and it's been quite fun.

Embedded has been consolidated into a bunch of huge multinationals like Texas Instruments, Qualcomm, etc. and they're doing alright financially. It's just a very innovation-lacking industry compared to software, so you have a chicken-egg problem with investors. It'd be like trying to start a CPU company now when we've got Intel, AMD, IBM, and maybe one or two more boutique makers teetering on the edge of bankruptcy constantly.

[StreamBright]

The ratio what matters not the number of devs available.

demand / supply

You might find 1000 devs with react but if the need is 10000 while there is 50 for 40 VxWorks devs then it explains the salary.

If you downvote pls. explain how supply and demand is not a thing for tech salaries.

[panpanna]

> If you downvote pls. explain how supply and demand is not a thing for tech salaries.

HN is starting to look like Reddit where people bury your comment if it conflicts with their views.

I feel downvote on HN should affect your own karma with, say, -100.

(I also have started seeing significant misuse of flagging on HN)

[StreamBright]

Exactly. People deny basic economics and down vote because of their feelings.

[lallysingh]

I agree with you, as ultimately an employer will have to raise wages to get/keep employees.

But it's not really enough to explain it because the engineers who do Vx could always just switch to a better paying platform.

I wonder if the low salary indicates that the Vx Devs at that low price aren't able to switch out. Either due to location, lack of ability, or some other capture.

[smolder]

It's not always trivial to change technologies job-wise. You may be perfectly confident in a technology you haven't used before professionally, but people doing hiring may think different. I have been screened out for "only" having hobby/extracurricular experience with tech I'm fully qualified to use.

[StreamBright]

As anecdotal evidence I know many developers (including myself) who refuse to touch frontend. It still stands, supply and demand rules apply. Does not matter why the supply is low or the demand is high.

[cardiffspaceman]

I am an embedded developer using primarily C++, but I've done a ton with Javascript. I don't know why an embedded developer would refuse to touch frontend, other than a suspicion it would fit the stereotypes of database applications (CRUD, tables). The embedded UI's I've worked with the most were not in HTML but we have done demos using WebKit and Angular or GWT. Given the asynchronous constructs it takes to keep our product running properly, the asynch in Angular was not an issue. We also had an in-house test tool that we ran for over a decade that used Javascript for the tests themselves. This was not asynchronous but we ended up with several engineers who knew Javascript's scope peculiarities and unit test requirements because of it.

[nicoburns]

Yeah, frontend dev is it's own beast. I've seen competent C++ devs at a complete loss when facing JavaScript. Especially if they've never done any asynchronous programming before.

[FpUser]

I am not entirely sure what qualifies for being "competent C++ devs". My thinking goes that it is kinda hard to imagine competent developer in general (never mind C++) not having knowledge about async patterns.

[StreamBright]

I have done async in Clojure, Erlang (Elixir) and Rust but never touched Javascript (wat?). Not sure you can attribute async to frontend development. Erlang predates Javascript by almost 10 years.

[sansnomme]

Unlike Erlang/Go/certain Clojure libs, the JavaScript flavor of async is primarily around red-blue functions. A lot of stuff is explicit and not handled by the runtime. A single async keyword can easily taint the entire codebase.

See: https://news.ycombinator.com/item?id=8984648

More importantly, frontend is extremely demanding in terms of UI. While for e.g. QT, WxWidgets, native GUI toolkits, it is perfectly acceptable for things to look the same. On the modern web, every significant app is expected to have a "design language" and not look like each other. On the other hand, HTML by default isn't very stateful, sure it has some stateful components like checkboxes but in general you have to store the state in code. And every code base needs a custom UI. You end up with a state space explosion. That is also why frontend frameworks these days are trying to force things into finite state machines i.e. Redux, Elm architecture. Whatever term you prefer. Half of your UI codebase often have absolutely nothing to do with business logic and instead animations and user experience code to make things feel "slick". A ASP.NET WebForms just ain't going to cut it if you wanna build a SaaS. Sad because tons of money have been wasted on essentially marginal improvements on user experience, but good if you are the consumer of course.

[nicoburns]

> Not sure you can attribute async to frontend development.

Not trying to do that. But (unlike C++), you have to understand async to use JavaScript sensibly. If for whatever reason you haven't come across it another language, then you might struggle (esp. if you're coming from C or C++ where the skills required are very different).

[jmnicolas]

I'd guess that most VxWorks projects are for governments so the pay is not that great.

On the opposite side a private company that makes billions $$$ with its website should be more prone to pay its devs better.

[pnako]

Aside from the already mentioned supply/demand aspect, people working on things like that tend to do so for an entire career, so might be willing to accept a lower salary in exchange for stability.

The React developer will have to retrain and rebrand themselves as Bojombo developers in 2 weeks and as Klazoum Framework ninjas in 2 years.

[linuxlizard]

You got me. I just started learning React and your comment freaked me out. Oh, crap, I have to learn Bojombo now!?

[jlokier]

It's a nice idea, but it doesn't match my experience.

I find that almost every new embedded project comes with yet another vendor architecture, vendor-specific toolchain, documentation that's wrong, totally different peripherals and drivers, weird limitations, undocumented pipeline glitches, and so on. Even the CPU architectures and instruction sets are often different, if it's a DSP, or an SoC with DSP coprocessors glued on.

Basically, new things have to be learned often in that domain, at least with the jobs that come my way. For a while I got a bit anxious on new projects because I couldn't understand how to make seemingly broken tools do things the client wanted. Things like getting signal data into a DSP simulator when all the (GUI only!) buttons didn't do anything at all for example; but eventually I learned that it takes about 2 weeks on every new project of restraining the urge to swear, and than all the undocumented stuff, tools working differently than documentation says, secret header files and options, libraries that no client of the vendor has ever used and are full of empty function bodies or just broken, but sold to my client as if they are actually in mainstream use, weird memory models etc. starts to make sense and I'll be productive. So I just expect that on any new architecture project now, and it works out ok. Just budget 2 weeks of swearing-restraint at the beginning.

On front end webdev, there is infamously a huge amount of churn too. E.g. every time a new Webpack comes out I have to spend quite a while figuring out how to rewrite the config files for the new version for best results. WASM differs from asm.js; grid and flexbox are a new way to achieve what we used to do with floats, and there's an interesting new browser API each week.

But React hasn't change much in half a decade, and continues to be one of the main front-end frameworks. You could have learned React 4 years ago, and keep on using the same skills today and continue to be well paid if you're good at it. That pay is, indeed, at least 50% more than the pay offerred for low-level embedded, even though React work is way, way easier.

But remember, you can negotiate... Just taking a software dev position as an individual is not the route to the best pay in embedded hw/sw projects. If you can instead sell product development, where the client wants something built for them, and you build it, complete with project management, architecture, technology selection, supply chain, and subcontracting or setting up a project-specific company, then the scarcity of relevant skills works in your favour and the pricing is quite different - better. Those projects are great if you can get them, and sometimes a lot of fun.

[spamizbad]

This surprised me too. But I think it really boils down to there being both more demand and less available supply of React developers.

Embedded C/C++ development has been around much longer and is far more conservative in terms of adopting new technology. So they have a much larger labor pool of experienced developers who are going to be familiar with the core technologies of the job. Web shops migrated quickly to technologies like Angular and React, whereas the vast majority of C++ shops likely won't be adopting C++17 any time soon unless it's a smaller team's greenfield endeavor. I'd also argue there's less training necessary to go from, say, C++03 to C++17 than going from ES5 jQuery to ES6+ (or Typescript) React.

[tuckerpo]

Hard disagree on the C++03 -> C++17 bit. They're like two completely different languages.

[kitd]

Once again demonstrating the tenuous link between pay and skill. Industry and public impact are bigger influences.

[nicoburns]

I agree it's bizarre. I think it's as simple as there being a lot more demand for React developers. There are a lot of companies that want a web or mobile app. Comparatively few have embedded projects.

[closeparen]

I'd guess that the economic value on the table in industrial controls is just not that high. It's a mature, stable, slow-moving market. The growth parts have to do with "IoT" and "Cloud," not so much the embedded side.

Gaming has similar-ish needs but is famously oversupplied.

Quant finance pays really well, though.

[humanrebar]

Some kinds of company succeed and fail based on good engineering. Others have other competitive advantages.

Given the... mixed... reaction to large defense projects, it's possible that the defense contractors are not focusing on engineering per se.

[castratikron]

Yes, embedded is harder than web. But embedded can only scale to as many hardware units the company can sell, while web can in theory scale to everyone on the planet. Web pays more because of this.

[goatinaboat]

On the other hand people are comfortable paying actual money for hardware but expect websites to be free.

[ChuckNorris89]

Websites aren't free, they're mining your data just that average Joe doesn't know or doesn't care as long as it's free.