[Solvency]

I fundamentally don't understand why this project is seemingly so poorly documented? I've read articles describing the current teams having to still reverse engineer things by scrutinizing random documents and sketches as if it's still this very unknown system.

[xenadu02]

The project was not really designed to reach interstellar space originally. It was a somewhat rushed program to take advantage of the "Grand Tour" where the gas giants would all be aligned enough that a gravity-assist orbit could allow a spacecraft to fly by each of them. The alignment in question only happens every 175 years.

The interstellar portion was an add-on after the success of the original mission. The spacecraft were still operating so why not just keep operating them?

No one designing or building the probes imagined they'd still be operating 50+ years later. Even if they did space programs are constantly under threat from budget cuts so you can't exactly waste money on what-ifs for the future: you must focus on making the official mission succeed.

Also remember that the "desktop PC" was not yet a thing when this was designed. Engineers were drawing everything on paper. Storage space was extremely expensive in any case.

A modern program would (and most do!) put various versions of drawings in a version control system. Source would use an SCM so code history would be available. Even things like meeting notes would be available and searchable digitally.

[RajT88]

I have found that even the group I'm in being documentation-heavy, it's hard to read through everything and build the same context that another engineer has all stuffed into their head.

As you mention:

> available and searchable digitally.

Even with 100% everything written down, it takes a while to build up that context, and even carefully written documentation can have subtleties which send a consumer the wrong way.

Things are a lot easier than they used to be, but still not easy-easy.

[cduzz]

Documentation is extraordinarily difficult to create. You need to anticipate the potential questions and answer them and anticipate all the potential perspectives and answer them from that perspective.

What is enough? A reference describing all of a thing? The source code to a thing? The source code and build chain to make the thing? The source code, build chain, source code to the build chain to build the build chain to build the thing? The source code and the machine and the tape drive to read the tapes to build the ....

How much documentation for TOPS-10 would you need to implement wireguard on a toad? How much context do you need to even make that sentence even make any sense at all?

[RajT88]

Yep, and the example I like to go back to is API docs, or command line docs.

One I was reading this week had a field for "language", but doesn't clue you in to the valid values by providing a working example. English for example could be:

English

English (US)

en

en-US

eng

1033

409

...And that's just for the US dialect of English. And also doesn't tell you if any of this is case-sensitive or not! Sometimes yes, sometimes no.

[renhanxue]

>The project was not really designed to reach interstellar space originally.

Not only that, Voyager 2's flyby of Uranus and Neptune in the late 1980's was originally not intended either. As an aside, to this day Voyager 2 remains the only spacecraft to ever have visited either planet, and there are no firm plans for a followup, just some loose ideas about maybe launching something in the mid 2030's. Anyway, doing the Uranus/Neptune part of the mission required extensive software upgrades, which introduced Reed-Solomon error correction and image compression capabilities, among other things - the software as launched would not have been capable of a meaningful mission to Uranus and Neptune.

These days the Voyager program is lauded as an astonishing feat of engineering and one of the most inspiring science and engineering achievements of all time, but in the early 1970's the entire idea was NASA's red-headed stepchild and ended up cut down to a bare minimum. The Grand Tour mission concept (taking advantage of the extremely rare opportunity to visit Jupiter, Saturn, Uranus and Neptune in a single mission) was pitched as early as 1965, and by the early 1970's there were plans for launching four spacecraft, two bound for Jupiter-Saturn-Pluto and two bound for Jupiter-Uranus-Neptune. These were referred to as TOPS, Thermoelectric Outer Planets Spacecraft. But then people started complaining that it might cost a billion dollars (Apollo had cost $25 billion) and the whole thing became intensely political. Quoting from Voyager: The Grand Tour of Big Science (https://www.nasa.gov/history/SP-4219/Chapter11.html) by Andrew J. Butrica:

> Further complicating matters was Senator Clinton P. Anderson (D-NM), champion of the Los Alamos nuclear weapons laboratories and an enthusiast, until his retirement in 1973, of the development of a nuclear rocket engine called NERVA. As chair of both the Senate Aeronautical and Space Sciences Committee and the joint Atomic Energy Committee, Anderson provided NASA and the Atomic Energy Commission over $1.4 billion, about $500 million of which was spent in Los Alamos, for the development of the NERVA engine, which, Anderson held, was ideally suited for exploration of the outer planets, as well as for more advanced missions. Anderson worried that NASA and the OMB were shifting money from NERVA to fund Grand Tour. When the NASA budget came before Anderson's Aeronautical and Space Sciences Committee on May 12, 1971, his committee voted five to two to reduce Grand Tour's budget, while an amendment to increase NERVA funding passed. Werner von Braun worried that ardent congressional interest in NERVA would force a loss of Grand Tour in favor of a NERVA that had "no place to go."

> Meanwhile, NASA was trying to include Grand Tour as a new start in its 1972 fiscal budget. The Friedman report moved the Office of Management and Budget (OMB), in March 1971, to ask NASA to study simpler, less costly spacecraft alternatives to TOPS. The OMB also attempted to delay the Grand Tour start-up to fiscal 1973.

> (...)

> As NASA prepared its fiscal 1973 budget, rumors spread that the "budget pinch" was going to affect planetary programs deeply and that the reduction of the Grand Tour payload from 205 to 130 pounds was "a likely fact of life." Furthermore, Grand Tour now began to compete for funding with the latest NASA human program: the Space Shuttle. The fiscal 1973 budget request NASA submitted to the OMB on September 30, 1971 included both Grand Tour and the Space Shuttle. Throughout the autumn of 1971, several press reports presciently reported Grand Tour's vulnerability to a possible elimination or reduction. On December 11, 1971, James Fletcher, NASA administrator since April 27, 1971, learned from White House officials that Nixon was prepared to approve the shuttle program and that Nixon would not let NASA simultaneously fund the shuttle and the full TOPS Grand Tour in the 1973 budget or in subsequent fiscal years. Fletcher had to decide which was more important: Grand Tour or human flight.

Fletcher chose the shuttle, and what could be squeezed into the budget was an extension of the Mariner program to visit Jupiter and Saturn only. For budget reasons the spacecraft development was kept in-house at JPL rather than contracted out, and at JPL the dream of the full Grand Tour was still alive:

> Despite the limited aim of the Mariner Jupiter-Saturn, the mission had the Grand Tour launch window, that rare planetary alignment, and the engineers at JPL still had every intention of building a spacecraft that would last long enough to visit Uranus and Neptune. This intention was not emphasized; however, it was stated that a Mariner Jupiter-Saturn spacecraft might continue to Uranus if its mission at Saturn proved successful. The scientists working on the project knew that Mariner Jupiter-Saturn was going to go to Uranus and Neptune, too. As Bradford Smith, Leader of the Imaging Team, explained: "We understood at the time the enormous potential of this mission, that it could very well be one of the truly outstanding if not the most outstanding mission in the whole planetary exploration program."

Also for budget reasons, the spacecraft were limited to mostly reusing existing technology. Getting reprogrammable computers (without which they could never have been kept alive in the way they have) required a separate budget grant from Congress:

> Despite the reliance on extant technology, some money was set aside to develop new technology. Congress and the OMB approved an additional $7 million to the Mariner Jupiter-Saturn appropriation for scientific and technological enhancements. Part of that appropriation went to develop a reprogrammable onboard computer, which proved vital to maintaining Voyager 2 as a functioning observatory in space. Without properly functioning hardware, no science could be conducted.

In the end only Voyager 2 was launched on the full Grand Tour trajectory that would allow visiting all of Jupiter, Saturn, Uranus and Neptune; Voyager 1 was launched on an easier and much faster trajectory that would take it only to Jupiter and Saturn. Even then, the official decision to extend the Voyager 2 mission to Uranus was only approved in 1980.

Human spaceflight and its enormous appetite for money has always been a huge threat to actually exploring the solar system beyond Earth orbit, and we should be very glad we got even the very diminished Voyager program that exists today.

[0cf8612b2e1e]

Shouldn’t this software archeology have been done decades ago?

[Someone]

What would be your argument for getting budget to do that?

There are zillions of ways this hardware can break down. You can’t predict which ones you’ll have to handle, or whether there will be a way to recover from them. If you started researching this 30 years ago, and then something had killed this thing 20 years ago, that would be wasted effort.

Also, in the early years, they still could ask the original engineers, and even lacking those, there likely were engineers who hadn’t worked on this specific hardware but were somewhat familiar with this kind of hardware.

[xenadu02]

By then it was too late. Computer tapes were so expensive they were regularly overwritten after a certain number of years.

Cheap storage is one of the most fundamental and under-appreciated game-changers.

Another under-appreciated changes is standardization of computer architectures and formats. 8-bit bytes. ASCII/UTF-8. Even media formats which all largely follow the QuickTime style of different streams and atoms and so on even if the codecs change.

In the 1970s none of this was true and everything was (by our standards) completely bespoke.

[KineticLensman]

I remember reading about the inquiry into the UK RAF Nimrod aircraft that came down in Afghanistan in 2006 killing its crew of 14 [0]. A significant finding was that recovering the design history and maintenance records involved trawling a massive number of filing cabinets / cardboard boxes scattered in sites across the UK, and was a significant cause of the missed opportunities to uncover the design flaws and near misses that preceded the crash. (long story short: internal fuel leak near a very hot exhaust pipe)

[0] https://en.wikipedia.org/wiki/2006_Royal_Air_Force_Nimrod_cr...

[TheCondor]

When was the first source code control system released? SCCS was like 1973 and the Voyager code was probably pretty much buttoned by then; with whatever practices that they thought was stable state of the art practices at the time. I imaging that this was a collection of "golden tapes" or something. Now the concept of revision control seems pretty self validating but you're talking about undergoing a culture change on your software team, pretty close to launch.

Then the voyager hardware was bespoke.

We just live in a different world now, they didn't know how to do software engineering like we do. They were just figuring it out. I really don't know the history of it but Voyager systems may have been produced on punchcard. Like the original source code might be physical for parts of the system.

[renhanxue]

The Voyager software was updated repeatedly and significantly in flight while the spacecraft were still in their early years. They were very intentionally designed to be patched over the course of the mission. The software as launched was not capable of a meaningful mission beyond Saturn, because for budget reasons that was officially not on the cards at launch (the Voyager name came very late; the program was officially "Mariner Jupiter-Saturn" for a long time). Features like image compression were added in the early 1980's, after Voyager 2's mission extension to Uranus and Neptune had been approved.

Without any inside information on the program, I would expect that a lot of development has been done more or less ad-hoc over the decades, as budgets have allowed and operational requirements demanded.

[zilti]

> they didn't know how to do software engineering like we do.

Yes, luckily. If they did, it would have broken after four years, and would have needed a second nuclear battery due to the inefficient code.

[toast0]

If they have random documents to scrutinize, doesn't that mean that it's documented?

When I work on undocumented systems, it's because someone wrote code with no design docs, no (retained) notes, no requirements, no specs, and it's been determined that it doesn't work right. All I have is the code, and current observations.

[liquidpele]

Haha look at this guy, thinking you can trust the docs. ;)

[Johnny555]

I'd imagine that a lot of the undocumented stuff was for things that were obvious to an engineer at the time -- I doubt many engineers working on it at the time thought that the probe would outlive their own lifetime.

I've run into lots of software comments in legacy code that refer to features or systems the company used to have that were deprecated years ago and are nearly meaningless today. Knowing that a flag was set to match the flags from the WOPR sytem isn't that useful when WOPR hasn't existed since before I joined the company.

[anigbrowl]

At the time engineers imagined it having a relatively short operating life, and (imho) also thought we would be putting out a lot more probes. During the Cold War space exploration provided both prestige and a technological proving ground. After the USSR fell, a lot of Congressional enthusiasm for space projects diminished and management became increasingly risk-averse because budgets were much tighter.

[mardifoufs]

I think it's because it was more of an awesome moonshot project that didn't really fit into NASA's shifting goals at the time and with the shuttle overshadowing everything else that happened then. No one was really expecting this much from the probes

[nonethewiser]

I imagine its a problem of distance, feedback, and lack of any analogous test environment.