[sherry-sherry]

From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO ---

ABC: Can you tell us why the system's gone down?

Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.

ABC: Do they know what happened though? Who's there the team that you've got working on it?

Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.

---

Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )

[grecy]

I worked for a large Telco (not Optus) and was constantly shocked by how little the management knew about anything.

They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it).

The Director of IT had me setup her home wifi because she had no idea what DHCP was.

The VP of "IT/Internet" literally had other people make her Powerpoints because she couldn't. She had never done anything more on a computer than send emails.

It was shocking, but also really eye-opening to realize how important business acumen and playing the political game were.

[zadokshi]

Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know. Especially in these larger types of companies.

[gottorf]

> they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know.

Well, technical competence and expertise does not necessarily translate to managerial competence and expertise. It's two distinct skill sets. Getting machines to work is very different than getting people to work.

[Sohcahtoa82]

> Well, technical competence and expertise does not necessarily translate to managerial competence and expertise.

This can't be overstated. The best engineer could be the worst manager.

That said, managers should have at least some level of technical knowledge on what they're managing. I'm a security engineer that once worked under a CISO who had very little technical knowledge and it showed. Having to explain things to him multiple times got frustrating. We're talking on the level of "Hey, our nmap scans are showing we have port 443 open on our web app. We should have all ports closed." and then trying to explain why that would make our web app completely inaccessible.

[grecy]

> Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know. Especially in these larger types of companies.

100% spot on, I couldn't agree more. That was the biggest thing I learned during my time there. It was painful at times, but a very important lesson to learn.

[rstuart4133]

> Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain.

Being a technical person myself, and working in a small company most of my life I was blissfully unaware and then shocked. I guess it's true that closeted existence did leave me blind. But I don't think that is a consequence of being technical or an engineer - it was just because the smaller companies I preferred to work for would go broke if they made dumb decisions or failed to learn. The shock came from the small company I worked for being taken over by a huge, word wide company. And then watching that huge company roll over all their internet links to Optus.

All technical people get to know what it's like to deal with the big two telco's in Australia. They all eventually figure out the smaller ones provide much better service and a sharper price, and then avoid the big ones like the plague. So we knew what would happen when we went to Optus, and it panned out pretty much as expected. That's perhaps an exaggeration, as them leaking most of their customer credentials onto the internet wasn't "expected", but then level of engineering incompetence that lead to the leak wasn't a surprise either.

What was a shock was seeing first hand how Optus won the business. All the rules that engineers live by, like basing decisions on empirical data, "trust but verify" and so on weren't part of it. Instead it was smooth talking salesman, mandatorily weekly meetings with implementation teams who contribution seemed to be to presenting spreadsheets proclaiming how much progress that had made, paid trips to holiday destinations for upper managers and cases of wine being sent to lower managers. Later came the lies, and when we called out the lies and dug up older versions of spreadsheets to prove things weren't going a planned white anting. The white anting worked - we were threatened with dismissal.

Those tactics only work for so long of course, and eventually the exec's that threatened us with dismissal ordered us to get rid of Optus, at any cost. Fawning turned into literal hatred. They could away such displays of corporate disloyalty only because by that time all the managers who championed the charge to Optus had gone.

But despite that experience, they moved to Telstra. Telstra is the other large telco in Australia, and is just as bad. I had left by then, but I assume the decision was made by a new cohort of managers who will also be gone in a few years.

At this point I'm no longer blind, just flummoxed.

[usr1106]

Not a surprise the CEO does not understand why their network is down. Many CEOs of companies of that size think business is moving money around, doing acquisitions etc. They have no clue how their production works or where it could be vulnerable. The information they get has been filtered through several levels of management, the cynic would say through several levels of of increasing incompetence in production matters.

[hilbert42]

Optus was down in most of Sydney for over 10 hours. Some country/outlying areas were back online a little before then.

In response to hours of silence from Optus executives (in that they weren't able to string two words together about the technical issues or the source of the fault) one caller to ABC Local Radio in Sydney asked whether the fault was in Australia or in Singapore which was an excellent question as Optus is owned by Singtel (Singapore Telecommunications Limited).

The logic behind the call was to ask about how much of the Optus infrastructure actually exists in Singapore. If, say, much of the technical control is done from there then it may have been hard to quickly diagnose the problem with the system infrastructure split across two countries.

This is important because when Optus was sold to Singtel there was much concern about the security of its telecommunications if another country had access to user data, etc. At the time we were told the Australian Government was happy with the security arrangements that were put in place but many were not completely satisfied with the assurance.

Given the huge data breach that Optus experienced some months ago and that that matter has not been resolved by any stretch there will have to be a fully-fledged inquiry into the failure and hopefully it will get to the bottom of how Singtel actually runs Optus.

That said, some background is needed for overseas readers.

Australia has a long, long history of screwing up telecommunications, radio and TV broadcasting not to mention spectrum management. Too involed to describe here in detail but briefly the stuff-ups and political shenanigans would fill volumes and they go right back to the Wireless Telegraphy Act of 1905 and the Postmaster General's Act. The Govt. PMG's Dept. had exclusive monopoly over the phone network for nearly a century (like Ma Bell but govt. owned). It later became Telecom Australia but still govt. owned. Still later the then Government sold it off and it's now a public corp under the name Telstra.

To maximize income from the Telstra sell-off the Government sold it off in two stages, and to make the sale attractive to investors it sold the telco complete—lock, stock and barrel without due consideration of the public interest. That is, instead of just selling off the exchanges and phone network infrastructure the Government also sold off all cableways and rights-of-way across the nation as a bundled deal. Also, it never put sufficiently tight regulations or conditions in place to protect customers and they've since been screwed rotten by all telcos—horrible deals, little or no service—for instance, witness today's Optus outage!

The sale of the cableways came back to bite the Government later when it needed to establish the NBN—National Broadband Network—as it had to buy back access to Telstra cableways which cost many billions—it's a huge scandal the Government somehow swept under the carpet. Also, the competition couldn't use the Telstra cableways, Optus had to completely install its new network onto power poles.

This first-class fuckup has now been 'amortized' into a surcharge that everyone across Australia has to pay for internet and phone. The extra billions have had to come from somewhere and it's the poor hapless consumer that has been paying the bill.

(In the past I've suggested some enterprising person do a PhD on the subject and in the process determine the extra per capita costs that every citizen is now paying over and above what he/she ought to have been paying had there been prudent management and less destructive politics.)

Incidentally, the NBN itself was further fucked up by politics, half-measures and technical compromises, it's story in itself. Just this week there's been an announcement that NBN network prices are to be increased.

Into this almighty shemozzle comes Telstra's competition, Vodafone and Optus. These telcos have also benefited from slack regulations, very poor consumer law and seemingly nonexistent technical oversight by Government.

Right, when it comes to telecommutations, Australia is the laughing stock of the world and its citizens have been taken for a very expensive ride.

[hilbert42]

Addendum. Well, as predicted, the Government announced it's holding an inquiry into Optus. That's less than 24 hours after the event and has to be a speed record for Govt. Such speed gives a good indication of how big this kerfuffle was and how bad the public's perception of Optus and Govt. is for allowing the failure to happen. The questions everyone is asking is why wasn't there a backup plan and why did the emergency phone numbers also fail as they are supposed to be protected (isolated) when such outages happen.

Many are saying Govt. is largely at fault and asleep at the wheel for not keeping stricter tabs on these telcos.

Given there are three major networks in Australia and a few minor ones and that only Optus went down the level of disruption was alarming. Essentially, much of the country was in turmoil for better part of a day.

If a comparatively small outage can cause such disruption to a country then just imagine the chaos that'll follow when the next Carrington Event hits us.