Hacker News new | ask | show | jobs
by SoftTalker 9 days ago
Imagine something like this happening halfway to Mars and zero chance of escaping, getting any help or parts sent to you.
10 comments

Recently started an embedded hardware/software job. Shipping firmware to the manufacturer feels like that for the device classes that have no internet.
My first week on the job they told me they're about to manufacture 20k units and can you please fix this bug in the firmware by Friday?

I've never shipped anything to real customers in the wild before, so let me tell you how insanely stressed I was to open the firmware and find a 10k lines of C contained entirely within a single switch statement. I think they used some no-code tool to graphically design a state machine then plopped the generated code straight into the device.

Nearly the same experience. Had to fix an issue in a boot loader. It came down to improper setup of the memory controllers ECC engine. It would correct and ignore a single fault. If you managed to get two faults it would raise an exception that was not handled and the boot would fail. For the customer this meant that a reboot might randomly brick the unit until you go in and manually power cycle it.

Just convincing them that their problem boiled down to a single incorrect bit was difficult enough but then having to, in a day, build and successfully operate a test harness to prove the fix worked was the real stress.

I do not miss embedded engineering.

Anything special you noticed about the deployment processes involved with that versus more typical software engineering work?
Software can be updated and patched, even if you have to manually email customers a bespoke exe that pokes bytes into a compiled dll.

Generally firmware can't be updated by the end user because there is physically no way to do so without returning the hardware. (Unless an update mechanism is specifically implemented in hardware, obv)

Pucker factor goes way up because if you ship a bug, there's no way back. If you aren't careful, you can break physical devices which can have consequences anywhere from thousands of RMAs to burning down a user's house depending on the hardware and how bad you fucked up.

The deployment process itself is about the same. Tests and more tests, including testing on prototype and/or pre-production units. Hardware testing can get wild depending on application, but I don't think any SWE would find it too surprising. Then you email a binary to your manufacturer and pray

I can’t quite imagine, even shipping on prem stuff is much harder than the cloud. Especially when people can mess with stuff
The Zvezda module has been in orbit since July 2000.

I don't think any crewed interplanetary mission is going to last that long for the foreseeable future.

There have been quite a few instances over the years of Zvezda requiring repairs that would have been impossible without resupply from Earth.

I suppose that they were counting on the capability to resupply, otherwise they might have carried more contingencies from launch, but still.

Sort of like what happened on the Apollo 13 mission in 1970. Engineers on the ground were able to devise a makeshift fix to adapt the control module airscrubber filters to fit the lunar module so the astronauts could shelter in the LM for several days before getting back into the CM and coming home.
Yeah I was thinking about that, the big difference being that you are months out instead of hours/days, if a return to Earth is even possible.
I'm not sure distance matters. They're still stranded with virtually no possibility of rescue from the ground. Apollo 13 was extremely lucky that the hull wasn't breached, the spacecraft could still be controlled, that some very smart guys on the ground were able to devise the fix using bits of stuff known to be on board, the filter could actually be made, and, most of all, that it worked.
Ideally your Mars transit vehicle hasn't been taking 90 minute heating and cooling cycles nonstop for 26 years.
Well one side will be facing the sun and the other will be facing the void, so there might be similar issues.
There'a maneuver called a "BBQ roll" where you basically set the craft to doing a barrel roll in order to prevent any one side from overheating. I image that could help some.
IIRC during transit you'd want as much mass between you and the Sun (as shielding), and as small of a cross section facing the Sun. Probably also to reduce heat reaching the propellants.

So in a cylindrical ship you'd want to have one end pointing at the Sun most of the trip. This is, of course, very different in effect on the hull compared to the repeated expansion and contraction of heating cycles.

That's not necessarily true. Even spaceships in LEO will perform temperature-driven rolls so as to distribute heat and radiation. I have to assume that long-term ships like interplanetary transport will do the same.
Interesting thought. Isn't it possible to design around this?

Surely this was considered when building the first modules.

Of course you can, but "needs to survive 26+ years" was very likely not part of the original design goals. The designers of the time probably wouldn't have expected the dysfunction to be so deep that 26 years later, only the Chinese can seem to stick to a plan.
You can design around a lot of stuff but what you encounter in orbit will ultimately laugh at that bandage and eat it away. AtOx, hard UV, and radiation levels you don't get on Earth just have their way with everything in orbit over time.

You don't get the AtOx going to mars but you have everything else which will utterly take its toll on a traveling craft.

There is less debris around on the way to mars and this is a known and worsening for the ISS due to its age.
A top (arguably, the top) metallurgist who studied previous failed parts told me it's corrosion of the Russian alloy used.

Corrosion is a hard problem in living quarters (ie moisture and salt) in space (sealed with no gravity)

In microgravity, everything gets everywhere. My mother worked on NASA funded research for diagnostic spit tests to determine chronic versus acute stress, which previously required blood draws, which are a less than optimal choice in space. It's all very stressful.
I was wondering about this as well. In theory, there are also some metals and compounds that react with each other with just simple contact which result in some kind of amalgamation which can result in disastrous structural loss. Veratassium recently did a video on this kind of effect[1]. Could this be happening here?

[1]: https://youtu.be/ksn5yrsC3Wg

Are you referring to galvanic corrosion? That's well understood and I'd hope not an issue in spacecraft manufacturing.
It seems as though the leaks are always in the Russian section? Perhaps this is why. Humans are the weak link. Damn breathers.
1980s Soviet metallurgy isnt known for corrosion resistance. Now the whole skin is made of that metal.

Don't forget the module we're talking about was built in the 80s originally intended for MIR 2. It's been in service since 2000.

Debris from what? Satellite debris get in that orbit?
Most of the things that will be a common danger (that is too small to track) are tiny pieces of stuff. Think paint chips and sand grain sized objects. These can be from things that came off rockets and ships, and things we've left behind like experiments and satellites. When these tiny things intercept you at many kilometers per second it can be dramatic.

Anything larger, say a lost screw driver, would punch thru the ISS like it wasn't even there leading to some ugly consequences.

I did an internship at NASA. What they told me is that anything larger than a golf ball they track while anything smaller than, I think they said a penny, is too small to do damage. The problem is debris that's in between the two. In that case they only get a relatively short warning (it's been a while but I think it was on the order of a couple hours).

The ISS can dodge debris by adjusting the height of its orbit.

Bits of spacecraft falling off (Challenger's windshield was famously cracked by a paint chip), debris from satellite collisions, even anti-satellite weapons tests.
Debris from space. Lots of rocks are constantly falling from space from all over. Sometimes they're big and make pretty lights in the sky as they fall, often they are practically invisible.
A bit of a tangent, but the fictional book "Children of Time" takes this to wild extremes. Really fun read
A little off-topic - the movie Stowaway (on Netflix) is a good movie about journey to Mars.
Seems like these structural integrity problems are always inside the Russian section. So if you're on a Russian mission to Mars, yes it would be reasonable to be worried. Otherwise this seems like a non-issue.
This is just not true. There have been leaks due to micrometers in just about every section of the ship at one point or another. A quick search pulls up examples of US modules having issues, especially around interfaces and seals. NASA had a whole investigation between 2018 and 2021 about the recurring issue.
This is just wrong. All serious issues that turned out to be safety concerns were in Russian modules. The 2018 leak you refer to here was in a Soyuz capsule and the 2021 leaks were in the Zvezda module (same place they are this time). In between there were also minor leaks in the Zvezda connection tunnel.
If you count the Soyuz leak, then the Boeing counts too! That was far more serious than anything you listed.

Two astronauts stranded for nine months taking the ISIS supplies intended for others. This is after they safely docked, which was considered risky at the time.

You brought it up. I have been talking about structural issues with long term core modules. And that is clearly a Russian issue.
I think you're confusing me with the OP, which in fairness I didn't read.

Nothing in the Russian space program in the last few decades have been as dangerous as Boeings little fiasco. Yes, the modules have long term problems, but they're built by the Russians because they have the most experience in space living quarters.

Look at space mission fatalities, the least Soviet/Russian one was in 1971 and that includes the 90s.

Thats 55 years

The US since then has had two shuttle disintegrations, the latest in 2003 when the US gave up launching astronauts for a few decades.

Space is hard.

The Boeing mission was scrubbed out of an abundance of caution. IIRC, nothing bad actually happened.
Maybe we can use the goop from those self sealing bike tires to have self sealing space station modules
They were never stranded.
>Otherwise this seems like a non-issue.

Except you forgot to mention an epic leak in Destiny just three years after it was attached to the ISS: "At its highest rate, the station was leaking about 5 pounds of air per day overboard." [0] Imagine that happening on the 4th year of American Mars mission.

Also, if you on American mission to Mars, it would be reasonable to worry about cooling system dying mid-flight requiring three spacewalks to fix it: "We'd lose cooling capability to half of the electronics on the U.S., European and Japanese part of the space station." [1]

[0] https://www.nbcnews.com/id/wbna3882962

[1] https://spaceflightnow.com/news/n1007/31station/

Unless your spacecraft is built by Boeing.

We had two astronauts stranded in space for the better part of a year just last year!

Ah yes, the well traveled and highly tested human mission to Mars.
The 10 non-Russian modules have been in vacuum for a quarter century and have done just fine despite facing more debris than in interplanetary space. So yes, this aspect is well tested. This stuff is literally part of the reason why the ISS exists in the first place.
The hubris of forgetfulness; to think that until Elon showed up the West couldn't even put a person in space anymore.

The Soyuz, the MIR, the human space records, the Venera program, closed cycle rockets, all have no equivalent in the West. Even their version of the shuttle was superior (it flew 100% autonomously).

I don't like Musk, but he single handedly saved the Western space programs.

I didn't realize Buran flew, and flew autonomously. Impressive for the times.
This sense of national pride based on long past achievements will always be bewildering to me. Do you really think a country that is actively engaged in a full scale open land war and whose economy is in shambles is able to maintain (much less build) a venerable space program? Elon might have saved the American tax payer from the senate launch system jobs program, but the majority of the global space industry is and always has been in the west. Russia has been an afterthought since the collapse of the Soviet Union. And it shows in everything they have done in space since.
Well, they managed to rebuild their launchpad ahead of schedule and launched this big boy not long ago:

https://www.space.com/space-exploration/launches-spacecraft/...

They've also got some new passenger jets certified and about to enter production (MC-21 and SU-100).

"This sense of national pride"

Im Italo-American. The closest I ever got to Russia was my cousin going to Moscow to, and I quote: "learn new things, like how to snort vodka"

It seems to me that you are projecting your dislike of the government of Russia into your evaluation of their engineering merits.

They landed drones on Venus, and on the Moon before Apollo 11

"Do you really think a country that is actively engaged in a full scale open land war and whose economy is in shambles is able to maintain (much less build) a venerable space program?"

Don't blame Russian space failures on the war.

Roskosmos was robbed blind by the likes of Dmitry Rogozin long before 2022. The Angara heavy launcher project has been started in the 1990s and still reminds me of Duke Nukem Forever. The Vostochnyi cosmodrome has been a black hole in red numbers for some 15 years etc. Things were "meh" even during the times when oil was 140 USD per barrel and Russia had no sanctions going against it.

Then you die and go into the history books.
They're not flying to Mars in a 30 year old Russian rust bucket so