| In the last 100 days, the MD Department of Information Technology (DoIT) has submitted $3,000,000 of ____emergency____ procurement requests for network monitoring, intrusion detection, and firewall assistance. All have been approved and funded. Sept 2021: - Splunk Emergency Professional Services - $800k - src: https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf Dec 2021: - Splunk Emergency Professional Services - $1.6m - Palo Alto Emergency Professional Services - $360k - Ciena Emergency Professional Services - $240k - src: https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf So...what do you think is going on at Maryland's Department of IT? ____TRULY SHAMEFUL INCIDENT COMMUNICATIONS____ Bad things happen in IT departments every day. Some can be anticipated, some even possibly prevented - but there will always exist the risk of data breach/attack. “Public officials have no higher responsibility than keeping the American people safe, and there is no greater threat to their safety than the cyber vulnerabilities of the systems that support our daily lives,” said Governor Hogan in November 2021. Secretary Leahy's commitment to the Governor and the people of Maryland is to lead the state's IT resources in taking all reasonable measures to protect against such risks. I do not yet know if this situation could have been reasonably prevented or if it is due to MD DoIT negligence/oversight. The flurry of emergency procurement requests suggests awareness and good intentions. And yet - here we are... What is clear to me, however, is that Secretary Leahy and his directs have ___failed to effectively manage communications___ throughout the response and remediation effort. Most damning, in my opinion, is the use of the incident report as a venue for high-fiving and taking victory laps: > Because of the state’s aggressive cybersecurity strategy, and the use of MD THINK and other cloud-based services, many of the department’s core functions were not affected. > src: https://health.maryland.gov/incidentupdate/Pages/default.asp... Are you serious? It's been over two weeks. And it's clear from your report that you're nowhere close to understanding what happened, let alone fixing it. ____FUNDAMENTAL MISUNDERSTANDING OF INCIDENT IMPACT____ Whoever approved this for publication is oblivious to the downstream, rippling effects this incident continues to have: - At the time of writing, MD's local public health departments have been ___fighting a war blindfolded___ for 16 days. - The impact has rippled ___beyond MD's state borders___, impacting CDC data feeds as well. Hospitalizations (which are reported by each hospital network directly to the CDC) are trending upward while new cases (reported by each state's IIS to the CDC) have flatlined: https://imgur.com/gallery/k2gG5GS. ____AND WHAT THE HECK IS MD THINK?____ MD Think is Maryland's cloud application platform. The state, along with Deloitte and an army of subcontractors, has been working on migrating legacy applications to this new infrastructure since 2017. The report points to the success of MD THINK in isolating the impact away from core MD Health Department services. Is MD Think really a success? Is this a case of what happens when legacy applications are left outside to rot in the sun? No. It turns out that MD Think has been a nightmare since it's inception. A series of program failures that jeopardized open enrollment for medicaid recipients led to a full audit of the MD DoIT in 2020. Here is what they had to say about MD THINK: > "...for one project (MD THINK), the estimated $314.1 million cost of completion had increased by $141.1 million (81.5 percent)...an explanation was not provided for the increase..." And on the topic of security: > "We identified 55 firewall rules that allowed traffic from any source to 71 unique network destinations as well as a small network segment within DoIT’s internal network without IDPS coverage..." > "...access to personally identifiable information (PII) for State vendors stored in the State’s Financial Management Information System (FMIS) was not adequately restricted, and the PII was accessible to thousands of State employees." > src: https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport... ____What About ImmuNET?____ A question that begs an answer is why ImmuNet, the state's immunization registry, is not included under this "titanium umbrella of protection?" And while there is a ton of publicly available paperwork about the scope of MD THINK, including the RFP, Deloitte's response, and transcripts of hearings to explore the audit report's findings, ImmuNet is not mentioned. Not once. Please keep in mind - ImmuNet is not some podunk, state government hobby project. It is __law__ that Maryland retain vaccination records for its residents. School systems, employers, health providers, and the CDC all rely on Immunization Information Systems (IIS) for ground truth vaccination records. It is the public health equivalent of DMV. ____PASSION DISCLAIMER____ The lion's share of my time this year has been spent working with state and local COVID vaccination programs within upstate New York. I have seen firsthand how important accurate, timely test and vaccination data is to how local public health departments coordinate response. It was infuriating to see the MD DoIT downplay disruptions to COVID related data feeds. We are going into year three of a war that demands accurate, timely data to inform decision making. What the hell have you guys been doing for two weeks?! The country's public health departments are filled with the true unsung heroes of the pandemic. They're underpaid, exhausted, afraid, and constantly under attack - yet they're our last (and in many ways, only) line of defense. SO... To the three people that made it all the way to the end of this post - I ask just one thing: IF YOU EVER FIND YOURSELF INVOLVED IN A FUCKUP THAT IMPACTS THE DEPARTMENT OF PUBLIC HEALTH: - RULE 1: Apologize and fix it. Now. With the kindness, urgency, and empathy a mission controller would show a stranded astronaut--Proceed to RULE 6. - RULE 2: If the problem is not your fault--return to RULE 1. - RULE 3: If the problem isn't even real--return to RULE 1. - RULE 4: If they are blaming you for a problem they created on their own--return to RULE 1. - RULE 5: If the problem is related to Microsoft Access, FoxPro, or any other technology you swore an oath to avoid--return to RULE 1. - RULE 6: The final rule. Do not downplay the importance of data and tools public health resources rely upon. And NEVER, EVER respond to the problem by boasting about all the great technical achievements made everywhere BUT the underfunded, under-appreciated public health department. |
I read through those procurement requests. The reasons they cite in the remarks are interesting.
Sept 2021, Splunk:
> The staff providing the Splunk support services resigned as of March 31, 2021, and the contractor has been unable to replace them. The Department requested these support services through another contract vehicle, but that contractor was unable to source qualified and competent candidates in the appropriate labor categories and with the rates provided in the contract.
Dec 2021, Palo Alto Firewalls:
> Although the Department has two contract vehicles available to staff this need, neither contractor has been successful in staffing the role. One contractor who once provided the services lost three of their four resources in the past 12 months. The other contractor has been unable to source a candidate with the requisite skills and experience in the appropriate labor categories and rates provided in the contract.
So people are quitting, and they can't find replacements. Wow, that makes me think we're seeing effects from the "Great Resignation". I wonder what the "rates provided in the contract" are.