Hacker News new | ask | show | jobs
by throwaway3306a 692 days ago
I have managed IT departments that managed hundreds of locations and thousands of computers running Windows XP and Windows Server 2003, no cloud at all. And I went through several similar outages (similar in impact on our operations, not cause or impact on others). Our first priority was to get the critical computers that operated machinery running - we did that hours (1-2) after the problem started. Then we played around with the servers and network for few weeks - but critical stuff was operable, albeit with lesser capacity and efficiency.

And we were managing forests and waterways, not hospitals and human lives.

1 comments

That's all fine, but this time, no one could get those computers back up in the first few hours, since they were stuck in a boot loop. Plus, systems like hospitals had to be running all that time. Plus, at the scale this outage is reported to be - banks, stores, factories, phones, emergency services, CNC machines, networking, aircon - I imagine everyone was confused and trying to figure out if anything works.

I'm happy nothing significant was hit over here in Poland; reading the main HN thread on the outage feels like reading war reports.

If it's stuck in a boot loop, the first thing I do is call the local admins and tell them to take a fresh SSD and a Windows installation USB drive with them. Plug the new SSD, reinstall the OS and copy the files from the old one. Computer running in less than an hour.

That's literally what we did to restart our forest logging machinery. Are human lives less critical than that?

You might consider that things have changed in the past 20 years. Also that medicine operates differently than forest logging.
Things haven't changed in IT so much. I am not in ICT management anymore, but I write software for the modern enterprise systems and networks - I'm reasonably up to date.

Ad medicine - hence my question, I'd really like to know what's the blocker. So far it seems the blocker is bad IT management, regulation and liability, not impossibility to perform the treatment.

Your answers indicate that you have not worked in n environment heavily dependent on ever shifting physical world logistics. You might try talking to some coordinators on the ground of a hospital, rescue center, consteuction site, theme park, or military operation for insight.
I talked to people in charge of the operations on a daily basis for years. I really don't think these considerations have changed that much since my times of leadership of an entire department managing just that.