Hacker News new | ask | show | jobs
by rzimmerman 660 days ago
I worked for about a year with a consulting firm that handled "Y2K compliance". Unlike this Andersen exercise in legal face-saving, it was a real job. Big companies hired us to do a full inventory of their site equipment (this included manufacturing plants, Pharma stuff) and go line by line with their vendors and figure out which components had known Y2K issues, which had not been tested at all, and which ones were fine/had simple fixes. We helped them replace and fix what needed to be fixed.

Y2K was a real problem. The end-of-the-world blackouts + planes falling from the sky was sensationalism, but there were real issues and most of them got fixed. Not trying to take away from this very interesting story of corrupt cronyism, but there were serious people dealing with serious problems out there. "Remember Y2K? Nothing happened!" is a super toxic lesson to take away from a rare success where people came together and fixed something instead of firefighting disasters.

11 comments

I like the phrase "Self-unfulfilling prophecy", for when a prediction that is acted-upon often invalidates its future.

There's also this annoying catch-22 which may be familiar to IT staff:

1. Things go wrong: "What do we even pay you for?"

2. Things go right: "What do we even pay you for?"

And this is why everything goes in writing.
...and 24 years later, after the paperwork has been filed away, someone will still write that the problem never existed. Y2K minimization and anti-vaxx sentimental are 2 symptoms of problems solved so successfully, the magnitude of the problem disappears from the collective consciousness.
and then 2038 unix epoc will happen and this time everything is computerized and as cloudstrike showed no one bothers to test any of it anymore. Oops
Place I used to work had a cycle of "Everything is working. We don't need quite this much IT staff." and "Everything is broken. Clearly we need more IT staff."

How the people in charge of this stuff never noticed the cycle is beyond me.

I've seen this phenomenon play out multiple times in my professional career, where a considerable amount of effort goes into creating a robust system only for the effort to be minimized by management due to its stability. Somehow preventing a plane from crashing is not as valuable as digging through the ashes.
I have made a career if stabilizing systems... but the secret is that they have to be pretty close to on fire before I am ever called in.
And unfortunately sometimes you have to just let the fire smolder before anyone cares enough to let you fix it.
This. Make problems ugly, any way you can.
A lot of jobs are like that. Accounting, legal and security work is only visible when it was bad.

Noone is being promoted for doing good job that prevented an unanticipated disaster from happening.

In agreement with your overall point, accounting and legal is different though.

For accounting, it's not a simple cost center, and is at the front line to show the numbers. They can say how much it will cost to not comply with a rule, or how much they saved by creativity or ingenuity. Being that close to the money is a tremendous advantage.

Legal is more distant, but there's a clear scale of how much is on the line. When you review a contract, it's pretty clear what’s at stake if legal work is botched.

That's my main takeaway: if you care about money, you need to be as close to it as possible. At the same skill level, dealing with user security or financial transaction security won't pay the same.

That's why presentation and "sales" type skills can be useful. A bit of doom-mongering internal PR about the problem, then present the solution. Don't just solve it quietly.
„there is no glory in prevention”.
> "Remember Y2K? Nothing happened!" is a super toxic lesson to take away […]

See perhaps:

> Normalcy bias, or normality bias, is a cognitive bias which leads people to disbelieve or minimize threat warnings.[1] Consequently, individuals underestimate the likelihood of a disaster, when it might affect them, and its potential adverse effects.[2] The normalcy bias causes many people to prepare inadequately for natural disasters, market crashes, and calamities caused by human error. About 80% of people reportedly display normalcy bias during a disaster.[3]

* https://en.wikipedia.org/wiki/Normalcy_bias

Also maybe:

> Optimism bias or optimistic bias is a cognitive bias that causes someone to believe that they themselves are less likely to experience a negative event. It is also known as unrealistic optimism or comparative optimism.

* https://en.wikipedia.org/wiki/Optimism_bias

I got my first IT job doing Y2k compliance. About 20% of our systems broke with the date changed to 01/01/2000, including the PABX (continual reboot cycle) and the sales leads system which would crash every few minutes.
> "Remember Y2K? Nothing happened!" is a super toxic lesson to take away from a rare success where people came together and fixed something instead of firefighting disasters.

My cynicism about Y2K comes from the fact that there were a lot of snarky articles written about how certain countries or companies were not Y2K ready but nothing bad seemed to happen to those countries either. It seems like a natural experiment was conducted and the results indicate there was no correlation between good outcomes and the work done to be Y2K ready.

I have no doubt that the armies of consultants did fix real issues but anyone working in software knows there is a never ending set of things to fix. The real issue is whether that work was necessary for the ongoing functioning of business or society.

"but nothing bad seemed to happen to those countries either."

Bad things still happened everywhere, despite all our efforts. How bad depends on your perspective.

Several people suffered a bizarre form of resurrection, which normally, Christians would be all over it and jolly excited. Pensions suddenly started paying out, tax bills became due from people long dead. If you were not a relative of one of those people it did not affect you and if you read about it, you'd have perhaps said "typical" and got on with life.

Some devices just went a bit weird and needed turning off and on again. Who cares or even noticed? Someone did but again, you did not hear about those.

I spent quite a while patching NetWare boxes and applying some very shaky patches to Windows workstations. To be honest, back then, timezone changes were more frightening than Y2K - they happen twice a year and something would always crash or go wrong.

The sheer amount of stuff that was fixed was vast and I don't think your "countries that did and did not" thought experiment is valid. Especially as it is conducted without personal experience nor much beyond a bit of "ecce, fiat" blather.

Nowadays time is so easy to deal with. Oh, do you remember when MS fucked up certificates and Feb 29 a few years back?

Your examples make my point. Some bad things happened but not on a catastrophic level that warranted the level of investment that was put into Y2K projects.
Most of the companies I was familiar with then did not have enough time or resources to check for and resolve every problem, and these problems were very real. At some companies the engineers were given autonomy, authority, and effectively unlimited budget to do literally whatever was required to mitigate any publicly visible failures that occurred. We had a lot of backup plans to keep operations running, sometimes literally paper and pencil, when the inevitable failures occurred. A lot of companies were furiously faking it and throwing people at the problem.

I directly witnessed a few near catastrophic failures due to Y2K at different companies, literally company killers. We kept everything (barely) running long enough to shore up and address the failures without anyone noticing, partly because we had prepared to operate in the face of those failures since we knew there was no way to fix them beforehand. It was a tremendous decentralized propaganda coup. No one wanted to be the company that failed as a result, the potential liability alone was massive.

The idea that what was averted was minor is a pretty naive take. I was genuinely surprised that we actually managed to hold some of the disasters together long enough — out of sight and out of mind — to fix them without anyone noticing critical systems were offline for months. IT was a bit more glitchy, slow, and unavailable back then, so the excuses were more plausible.

> Some bad things happened but not on a catastrophic level that warranted the level of investment that was put into Y2K projects.

Possibly the catastrophic things were prioritized and fixed?

Do you have any data at all to support your claim?
When things got missed things went _badly_ wrong and that spurred businesses to take rapid action to respond.

The first "Y2K" bugs where when banks' computer systems started messing up the calculations of long date financial securities/mortgages - decades before the millennium. Closer to the time Supermarkets started junking food that had a post 1999 Best Before date. Those were company ending problems if not fixed and so got overwhelming and rapid focus.

"... a lot of snarky articles written about how certain countries or companies were not Y2K ready..." I know you're talking about articles written after Jan 1, 2000. But there were a lot of articles written before then that were Jeremiah doomsday articles, so the snarky articles were reacting in part to equally wrong articles before then.

One article I recall in particular was in Scientific American some time in (IIRC) 1998 or early 1999. It prophesied (I use that word intentionally) that no matter how much money and effort was put into fixing the problem ahead of time, there would be all kinds of Bad Things happening on January 1. It called out in particular computers that were said to be unreachable, like hundreds of feet underwater on oil platforms. (Whether there actually were such computers, I don't know.) There was a sort of chart with the X-axis being effort spent on preventing the problem and the Y-axis being the scale of the resulting disaster. The graph leveled off while still in the "disaster" range, but still presented a clear message: "Give us more money and we can prevent catastrophes".

Somehow I haven't been able to find that article. Maybe SciAm suppressed it when the outcome turned out to be way short of a disaster.

There was also a TV (remember that?) news site or three that planned coverage beginning on midnight December 31 somewhere in Europe (Russia and China were off the map, I don't remember about Japan). Of course the news was that there was no news. (Yes, there were some computer programs that died or spit out junk, but nothing rising to the level of news.) I think it was an hour or two after midnight Eastern Time (US) that they ended the news cast.

Was there a Y2K problem? Of course. But it was largely taken care of before January 1, 2000, Y2K Jeremiahs notwithstanding.

I'm really curious how 2038 is going to play out.

I think it's either going to be a retirement plan for many who are young-ish IT people right now, or "optimists hoard gold, pessimists hoard cans and ammo" time with the pessimists being right. And a lot of this depends on how decision makers will remember Y2k.

Nobody is going to waste as much money on it as they did with Y2K and it's way more common for computers to actually use epoch time... but I think almost everything uses 64-bit time now and we're still more than a decade away.

(Don't reply with examples of things that use 32-bit time.)

I think 64 bit time is reasonably common in memory, but 32 bit time is common in serialized formats (both for storage and communication/exchange).
That narrative is only in the uneducated public. Every Y2K perfect of course documented its many findings and fixes.

In fact, the REASON Y2K got so much budget and attention were the early companies that started discovering the issues and alerted the others. Notables include IBM, General Motors, Citibank and American Express.

Agreed it was a nice success. We also did pretty well in paperless office, the ozone layer and acid rain, automobile and airplane safety, and the war on cancer, and now obesity and diabetes.

The public were not uneducated about this. If you remember how Y2K was presented to the public, it was ridiculously extreme - planes crashing, economies collapsing, etc. None of that happened, and not because all the bugs were fixed.

You can't fix all bugs, so if the consequences really were going to be catastrophic then you'd expect at least a handful of catastrophes to sneak through, but that didn't happen at all.

> None of that happened, and not because all the bugs were fixed.

No one is in a position to assert that. We have very little idea how fragile our civilization is. Perhaps it's pretty robust, and networks of interconnected problems (like Y2K) stand no chance of snowballing out of control. Or perhaps it's really, really fragile, and surprisingly little stands between us and a profound collapse.

It's very difficult to be certain, because it's such a complicated system, and one that we can't really test to destruction.

Would all the Y2K bugs have caused a widespread systematic failure if they'd gone un-fixed? Probably not... but maybe? Just like all low-probability, high impact risks, it's very hard for us to reason about.

How much money is it wise to spend on averting the risk of giant asteroid impacts? Hard to say. Probably more than you think, though.

A LOT happened, and was well reported, just not widely. It's more that the worst didn't happen, and that's what the media focused on.
The fact that it went so well is not evidence that no original issue existed. On the other hand, maybe it's evidence that we over-invested a bit into diminishing returns.

A perfectly fine-tuned response would have a little bit more to fix on January 1. Of course, expecting society to perfectly fine-tune the response for something poorly understood is hard.

Similar to the ozone crisis that was, and then wasn’t, but really was. https://www.unep.org/news-and-stories/story/rebuilding-ozone...
> Y2K was a real problem

Which only makes it more interesting. There are many takeaways one can have from this article, one of them is that:

- Problem X is serious.

- Y will address problem X

Is incomplete reasoning, or even an outright fallacy. Just because it's claimed that Y will address X doesn't mean it actually will.

Especially on high-stakes issues ("our business will collapse", security, safety) or emotive issues (social justice, security) this type of flawed reasoning seems to be a common problem.