[alephnerd]

It's generally professional services that set up these deployments at scale. MS's PS team is extremely competent and does push best practices in my experience. The issue is organizations that cheap out and decide to have an IT Service Desk guy manage everything from deployment to network architecture to security - these are extremely hard problems that require a large team of SME, not a single guy doing the best he can. City of Oakland is one of those kinds of organizations IME.

[nradov]

The lesson here is that the City of Oakland and similar organizations shouldn't be deploying such systems at all. They should lay off most of their IT staff and outsource their entire IT infrastructure to one of the large vendors who has the resources and technical competence to deal with advanced persistent security threats. Blaming the OS vendor won't accomplish anything.

[alephnerd]

Microsoft is an MSSP as well. And they are one of the multiple vendors the City of Oakland uses. But vendors can only do so much for organizations like Oakland as the final decision and implementation ends up getting stuck in red tape and bureaucratic hell between multiple disjointed teams.

[nradov]

What I'm proposing is that cities should outsource their entire IT infrastructure to a single vendor who runs the whole environment, including security. City employees shouldn't have any authority in these issues beyond vendor selection. I understand that might be politically difficult but what is the alternative? It isn't reasonable to expect city employees to have the skills and resources to defend against advanced persistent threats.

There is still room for city employees and other vendors to exert some control over higher level IT services and applications. But the core infrastructure needs to be under the control of a single competent vendor.

[PeterisP]

IMHO the core issue is a lack of resources - they can't afford an outsourced vendor that will do stuff properly just as much as they can't afford to do the same thing in-house. There's barely a budget to get hardware, and definitely not to deploy it properly; there's barely a budget to replace what dies of old age, and definitely not to do proper maintenance and updates.

[nradov]

Then the Oakland city council should make hard choices and reduce discretionary expenditures in other areas. That's a shame and it will hurt underprivileged city residents who are already struggling, but they need to face reality. I hate that we essentially all have to pay a "tax" to protect against IT security threats but what is the alternative? There doesn't seem to be a cheaper option that actually works.

[taeric]

I expect that pricing has made it so most all smaller places are these kinds of organizations.

And the incentives are to keep it that way. As long as MS's PS team can make more money from one whale of a customer than they can supporting local districts, expect that this will remain.

Such that I don't think it is excusable to say "if only they had paid the professional services."

[alephnerd]

You don't need professional services all the time, assuming you make sure to hire a large enough team of SMEs of your own, but a lot organization simply view tech as a cost center and try to spend almost nothing on it. That is the kind of issue City of Oakland faces (for everything btw - for example, starting salary for teachers in OUSD is $50k compared to $70-80k in the rest of the Bay Area). Also City of Oakland does 100% engage with Microsoft PS, but they don't have the requisite staff and budget to finish building out best practices.

[taeric]

I clearly don't know all the specifics of their case. Such that, if you have closer information, I am not trying to gaslight you on what you know.

I am concerned with "best practices" in our industry, though. Too many of them are not geared to wide adherence and have fantastically bad failure cases.

[alephnerd]

All good mate! There are some specifics I don't (and don't think I can) get into, but big picture, a lot of the core best practices in configuration management and security do work, the issue is whether you are able to hire people who can actually implement and understand WHY those practices are in place and how to iterate if said practices don't work. Most organizations across the globe have barely gotten a handle of ACLs and Security Groups, but evangelizing best practices for Endpoint Security, Cloud Security, OT Security, etc will take another 10-20 years simply because of inertia and the common sentiment that IT is a cost center.

Random think tanks doing thought leadership on CNN or at Brookings will jack themselves off to the notion of "online warfare" and whatnot, but those guys can barely type, let alone write cohesive policy.

I've been on both sides of this - both in the policy making world and in the private sector tech world - and cases like Oakland keep happening in a daily basis everywhere and will keep happening forever.

[908B64B197]

> Such that I don't think it is excusable to say "if only they had paid the professional services."

Would you apply the same logic to road infrastructure? Why hire those licensed engineers...

[taeric]

The target of the question is to the vendor of the road, though? If I pay to get roads installed and they counter that I should have paid for a more expensive process? Yeah, I'd say that is inexcusable. If they have strong reason to think that who they are selling to cannot maintain things correctly, they should consider not selling.