[firebacon]

The part that I find the most intriguing is "corrections can be made on the fly".

I can see how you would ensure reliability through proper requirements specification, a good software development process, separate independent implementations and extensive verification.

However, every time I read a popsci article about space flight software, they talk about this capability to push new code to the spacecraft while it is in flight.

I'm really curious to learn what this looks like in practice (technical details). Do they really have the ability to do an "ad-hoc" upload and execution of arbitrary code on these systems? If so, how are the ad-hoc programs tested and verified?

[marsRoverDev]

There is usually a piece of software running on the machine which basically just does this - allows you to command an image upload to the SSD, do a checksum of the file, then install it if all goes well. There is also usually a simpler version of the software on a redundant SSD or partition which the onboard computer will install if it detects that the software that is currently installed is malfunctioning.

My understanding is that some spacecraft launch with beta/alpha equivalent software. Correct me if I'm wrong, but I believe that the rovers do this, with simple software installed first, then more complicated versions installed once they know everything is working.

It's somewhat similar to updating your iphone, but instead you use a huge dish to do the transmission and the bitrate is pretty horrendous.

I'm going to need a definition of "ad-hoc" here; no-one "deploys straight to production" on a spacecraft. Any patches have to be thoroughly tested on simulators and models of the spacecraft on earth before they are transmitted.

[firebacon]

Thanks for the reply! So what you're saying is that it's just a "normal" over-the-air software update. I.e. you add some new functionality and then do a full system test of all functions of the software before replacing the entire image?

That makes sense, but is almost a bit disappointing. After all, that is exactly how it works for the boring systems here on earth. From various wired & co articles I had the impression that there was possibly something more; a mechanism that would allow users to send elaborate "commands" to the spacecraft to perform "ad-hoc" tasks at runtime. (What I mean by "ad-hoc" tasks are tasks that are unknown at the time of validation/testing of the software.)

[marsRoverDev]

Yes, we can send commands to the spacecraft once it's up there to do thing like modify memory or hard drive contents directly, turn on/off or command payloads and equipment. The full list is pretty exhaustive - anything you could want to be able to do, you can command manually. These things aren't 100% autonomous (though they have autonomous elements in the software).

There is also a way to send pre-programmed task lists to them which are executed sequentially, with delays if necessary.

That kind of thing is in the hands of operations, so it's not usually the job of the software team to test in the normal manner.

[firebacon]

Very interesting! Is this kind of command capability (e.g. ability to modify memory contents) something that is usually only available on "non-critical" subsystems, or would you generally expect to also find it on critical components, like the communication or navigation modules?

[marsRoverDev]

The ability to modify memory contents is pretty much universal; you can modify things like eeprom contents, the RAM, hard drive, etc. There is no differentiation between critical and non-critical; it's all just fairly critical.

Ground won't send telecommands to a spacecraft to modify a piece of memory without knowing exactly what they're doing first.

[Swizec]

It makes sense when you think anout it. These missions last months even years before a specific piece of software becomes useful because it’s for a specifi part of the mission.

Wouldn’t you want the benefit of those extra months to perfect the software?

[marsRoverDev]

I think that in some cases it's simply due to the launch schedules being more optimistic than reality; the hardware has to be done, but the software development doesn't necessarily have to stop once you launch the thing.

[natep]

In this case, the "corrections on the fly" refer to all of the real-time responses that the software makes without ground involvement. In the case of a solar limb sensor detecting the sun, the probe will abandon its data collection for that near approach, and go into an emergency response that has been made as straightforward and deterministic as possible, to maximize the chances of recovery for all single-fault and some double-fault scenarios.

To answer your question about software upload, the PSP has 3 redundant CPUs (primary, hot spare, backup spare), and each has multiple boot images. To upload software, the team uploads it to an inactive image of the backup spare CPU, promotes it to hot spare for long enough to collect the data it needs, reboots it into the new image, and then rotates it into the primary role, which is a seamless transition unless something goes wrong, and then the new hot spare takes over again within a second. Once they're sure the software is working, they can update the other CPUs. Before any of this, new software is tested on identical hardware set up on the ground with physics simulations.

See also, "Solar Probe Plus Flight Software - An Overview" from http://flightsoftware.jhuapl.edu/files/_site/workshops/2015/

[gdubs]

http://www.flownet.com/gat/jpl-lisp.html

[firebacon]

Thanks - that was exactly the kind of info I was looking for!

Amazing that they had the ability to just run ad-hoc LISP on the spacecraft. It appears their method to ensure safety in the face of arbitrary code execution was to divide up the spacecraft into isolation zones and run the parts that have a REPL on a non-essential CPU. From [1]:

> To protect the main DS-1 mission from possible misbehaviors of RA, the design included a “safety net” that allowed the RA experiment to be completely disabled with a single command, issued either from the ground or by on-board fault protection.

[1] https://ti.arc.nasa.gov/m/pub-archive/176h/0176%20(Havelund)...

[SomeHacker44]

From previous articles, remote updates seem to be a core part of spacecraft software/operating systems. I even recall one situation where a spacecraft had a REPL built in that was used to fix a problem (slowly) remotely! They also have multiple levels of operation and watchdog functionality. I have no direct experience with that beyond following news about spacecraft.

[firebacon]

Remote updates -- where you replace a full (sub)system -- are one thing, since you can always run the normal software validation procedure on the new version of the software. So an OTA update of a system (even in flight) does not sound like rocket science (yet)...

But: Once you include a REPL or another mechanism to push and execute arbitrary code "ad-hoc", I wonder how that could possibly be tested an validated? Surely as soon as you add the ability to run arbitrary code, there is no way of testing for all possible states of the system as part of the validation process?

In other words, how do you allow the user to push arbitrary code, but prevent them from putting the spacecraft into a condition from which it can not be recovered? The only way I could naively think of would be to only allow the user to push code to a completely isolated CPU that has a remote-reset functionality from the main/comms CPU.

Still, the popsci articles I read made it sound like there might be more to it. It would be excellent to find some first-hand accounts/sources on how this looks like in reality.

[CompuHacker]

One of the outer solar system probes had a radio that had enough built in logic to accept software updates and maneuvering commands independently of the two redundant on-board science computers.

Lights-out management indeed.