[sqeaky]

I did a few a few months of contracting at a major voting machine company. They make a significant portion of all US voting machines. They had 4 developer teams Firmware (C++ where I was), UI (web tech on a voting machine), poll book (java), and a web/support team. Before I was hired in a massive influx of contractors each team was something like 3~5 people, except UI which was a new team with the contractor hiring spree.

After the work was done, they shed nearly all the contractors and about half of their previous full time employees. Just quadrupled their staff to make a voting machine then fired them all.

They hired me as an "Embedded Software" on their Firmware team. It was a total shitshow we didn't have unit tests or CI. The new hires insisted on it and I spent a bunch of time maintaining a Jenkins setup for the team that really helped.

The pay wasn't great, a little less than defense contracting, which was a little less than insurance companies and slow finance companies.

If that is what most embedded development is like then I see why it is brings the average down.

[bombolo]

Ah yes I used to work at a california company as c++ developer.

There was no automated test whatsoever. QA department was just manually clicking things on a client that would connect to my c++ thing.

When I wrote a couple of unit tests I got told off because I was wasting time not doing features.

[jahewson]

Why waste time writing tests when you could be busy fixing bugs!?

[Remnant44]

Jokes on you, they probably didn't fix big either ;)

[bombolo]

Well the bug reports were like: "I clicked around and the UI froze/crashed"… no info on how to reproduce, no logs, nothing. Just that bit of information.

[ornornor]

So that’s what they teach in MBAs!

[theironhammer]

Best comment ever!

[sqeaky]

When was that? I am so glad that for the past 5~6 years every contract I have worked has had unit tests and for the past 10~12 every place has at least accepted their value.

The last time I actually had to argue for unit tests was in defense contracting and not for the team I was working on. Some idiot at a lunch-and-learn community thing tried to claim there was no short term gain from them and we had defined short term in months. He could not believe that unit tests can help the developer writing them and the help the team the very next sprint.

I hope he learned better or got forced out of tech.

[galangalalgol]

I have worked on codebases where full coverage was obtained using service level tests in a proper pipeline. If you couldn't add a service level test that got your pr coverage, then you were referred to yagni and told to stop doing stuff that wasn't part of your task. I was ok with that, it worked well, and the tests were both easier and faster to write. If the services had been too large maybe it would have fallen apart?

I have also worked on codebases where there were only tests at the level of collections of services. Those took too long to run to be useful. I want to push and see if I broke anything, not wait hours and hours. If a full system level test could complete in a few minutes I think I would be fine with that too. The key is checking your coverage to know you are testing stuff, and to write tests that matter. Coverage can be an antimetric.

[sqeaky]

> I have worked on codebases where full coverage was obtained using service level tests in a proper pipeline.

Sounds ideal to me. Add testing where it is cheap enough to justify, and maybe just a little more than you think you really need because they pay off so often.

If your mocks and fixtures get too big you might not be testing the code in question meaningfully.

Coverage and test quality/quantity need to scale with the associated risk. Perhaps "Flight planner for Air Force Weather" gets more testing than "Video game User Interface" and that's ok. Even in gaming, the engine team should have more tests than the game teams.

[DeathArrow]

If there are no unit tests, you can intentionally introduce some bugs, so you can have some easy fixes next sprint. Rinse and repeat.

[cmrdporcupine]

Sadly this sounds like a lot of crappy software shops generally, embedded or not.

[filoleg]

Yeah, but in real life scenarios, the difference in actual numbers, as opposed to percentages, matters.

Let's imagine that the split for all software shops is 80/20, with 80% being crappy, and 20% being decent. If there are 10 embedded software shops out there, it means there are only 2 decent embedded shops out there that an engineer can work at. Meanwhile, if there are 1000 non-embedded software shops, it means that there are 200 decent shops an engineer can work at.

This creates a wild disparity, even if the ratio of crappy to decent is exactly the same for all software shops in general.

[gridspy]

To build on your argument:

The 20% decent shops are retaining their engineers and only growing at a sustainable rate. Available new jobs are filled with a referral since every employee is constantly bragging to their friends. So they post few / no new jobs online.

The 80% crappy shops are shedding employees (turnover) and also poorly managed so they fire everyone and rehire later. Only the worst employees decide to remain during such a purge. So most new posted jobs (more than 80%) are for such companies.

Then the 80% crappy companies talk about their issues finding staff and you get articles complaining how hard it is to find XYZ employees (interns, C++, even supermarket staff). But the real problem is the company in question, not the industry as a whole.

[trap_goes_hot]

In real-life, engineers aren't just cogs in a wheel that are interchangeable, who can seek work in any organization. There is also a smaller number of people who can/want to do systems level/embedded programming.

[filoleg]

Yes, I agree with you. Which is why I explained that despite the overall ratio of crappy/decent shops might be the same for all software work areas, embedded devs are the ones who get the short straw.

[trap_goes_hot]

Not really, there are a small number of embedded devs competing for a small number of positions, so it all works out.

[ronanyeah]

> They make a significant portion of all US voting machines.

> Just quadrupled their staff to make a voting machine then fired them all.

Fascinating.

[sqeaky]

Just another project manager trying to hire enough people to make the project happen on time. I am in another one of those situation right now. Nothing to do with anything sensitive, just a team of 9 mothers trying to make a baby in 1 month.

[azinman2]

Sounds… problematic. What can you say about the state of security with it?

[sqeaky]

The code is quite secure, but the process and company are... typical processes and company people. Paper ballots and physical boxes are more secure if good practices are followed.

At one point I was tasked with shuffling the data layout on disk in real time to mitigate de-anonymization attacks. Security was real concern.

Crypto everywhere. The voted ballots were encrypted with keys generated and delivered immediately before the election. No networking by default. The end product had all the right things.

That said, no one had clearances, third party auditors were morons, and pay wasn't great. So if I were an attacker I would just try to bribe people to make the changes I want. Can't bribe a ballot box company to election tamper, because they just make boxes.

With all that effort they are still needless voting machines, they each count a few thousand votes and not all produce a physical paper trail. Because they have software and logic in them they need a constant chain of custody to make sure that the code we wrote is what is actually run.

Just use a box and paper, it is safer all the ways digital things suck. A precinct counting votes only needs to tally a few thousand ballots so it might take a team of people a hour or two, less time than to fix a potential technical problem.

And paper can more easily have bipartisan oversight and can have physical security measures that are impractical on a computer.

All that said I have no reason to believe our elections have been tampered with on a national level or that anyone other than a local republican may have used our machines to steal elections, even then no firm or even circumstantial evidence, just baseless suspicions and conspiracy theory level anomalies.

[speeder]

I am from Brazil. If you saw the news, the current president that just lost elections, been insisting for years, that elections here are untrustworthy.

Reason is simple: electronic voting machines with no logging, paper trail or anything. And the common people doesn't have permission to do penetration tests or read the entire source. All of it is proprietary and secretive with no public testing basically.

For years the now president, when he was still congressman, been trying to make a law where the voting machines will print the vote, and deposit on a box. This way people can count the votes printed not just trust the machine, but the government keep inventing reasons to not allow this, even when a law passed, judiciary struck it down.

Thus today people are protesting, seemly almost half of the country voted for him, the difference was tiny, they are protesting. The winner insists elections were fair, but how you prove it when the machines are proprietary and secret? How you prove it when they have no log of votes, and instead just print the totals? In a country full of corruption, and where the the mafia literally made a party to commemorate a specific person became chief election judge, how you trust nobody bribed the manufacturer or the programmers?

[sqeaky]

Most American voting machines print a ballot an let the voter review it, but not all. There have been some jurisdictions that have given up on that for reasons that seem bad and vague to me.

I think mandating that voting machines be open source is a good idea to me. Here in the US we have 3rd party auditing companies. Various US State and the Federal Government all have different testing/auditing labs that they have certified they trust. Then each voting machine company has to convince them that it is good to sell to the governments that trust them. The final build that the lab signs off on gets a cryptographic signature and the poll workers are supposed to check that it matches what they are given to run on their machines just before the setup their machines for voting.

Do Brazil have anything similar with auditors or inspectors? Or at least some crypto connecting the vendor to the polling locations?

[maleldil]

> Do Brazil have anything similar with auditors or inspectors?

Every year before elections, the government entity responsible for the voting machines invites hackers to run penetration tests [0].

> Or at least some crypto connecting the vendor to the polling locations?

The machines have no internet access at all.

[0, Portuguese]: https://www.tse.jus.br/comunicacao/noticias/2020/Abril/voce-...

[speeder]

Important to note the public test has lots of restrictions.

[josephg]

This is really interesting. Here in Australia we still use paper ballets for the lower house of parliament. I volunteered as a “scrutineer” for one of the parties, which let me go into the warehouse where the ballots were being counted and watch. As an scrutineer, you physically look over the shoulder of the person counting votes and double check their work. You can’t touch anything, but if you disagree with the vote, you can flag it. The voting slip gets physically sent to a committee somewhere for final judgement.

I highly recommend the experience if you’re Australian - it was very cool seeing democracy in action. I personally have a lot more faith in our system of voting after seeing it in action first hand.

That said, the senate votes are all typed into a computer by the election officials. It’s just too hard to do preferential voting by hand with ~200 candidates on the ballot.