[MrMetlHed]

Worked with a guy ten years ago that would obsessively read these reports from the Pentagon. Here's a couple of his stories about the matter: https://www.reuters.com/investigates/pentagon/ and https://www.reuters.com/investigates/special-report/usa-mari... (very old and I apologize for my horrific javascript at the time.)

It sounds like nothing has changed. Relevant to this crowd, maybe: "Most of the Cobol code the Pentagon uses for payroll and accounting was written in the 1960s, according to 2006 congressional testimony by Zack Gaddy, director of DFAS from May 2004 to September 2008."

"Wallace, the Army assistant deputy chief of staff, says the system has "seven million lines of Cobol code that hasn't been updated" in more than a dozen years, and significant parts of the code have been "corrupted." The older it gets, the harder it is to maintain. As DFAS itself said: "As time passes, the pool of Cobol expertise dwindles.""

[uticus]

Of course, devil's advocate would point out COBOL in 60s was written with a lot more focus on lasting in general. They didn't largely have the same general attitude of 'we iterate quickly with our script kiddies' as is prevalent today.

Not that all software from then was wonderful. Or that COBOL magically guarantees great code.

But, on the flip side, you can't say the code base isn't battle-hardened (to various literal degrees).

* edit: corrected sp

[ebiester]

I can build you a team using today's (quality: average B2B startup) programmers and today's methodologies that will write software for 40 years.

It will just cost twice as much as the team that iterates quickly. It will cost more because I will spend more money on business analysis up front. I will spend more money writing automated tests. Most importantly, I will maintain my own toolchain and libraries or pay someone to do so. Every piece of software deployed will have someone accountable to my business to maintain it.

It turns out that this is expensive. It's no more expensive than maintaining the COBOL code, mind you, and would likely give you better results than the equivalent COBOL code. Most companies will take their chances with less expensive software because that's what their competitors do.

About a decade ago, I saw one of the major systems you talk about up close. While they spent that amount of money for mission-critical systems, they didn't do nearly as good of a job on the average software in the system. I'm not convinced that any business is ready to go back to the battle hardened days.

[docandrew]

I wonder though whether the up-front costs incurred by your team would eventually be outweighed by the long-term costs of the quick team.

[ebiester]

I am not convinced that it would. The quick team will get to learn three times as much and adapt to the actual customer need. The cost of change in a system built with stability as a quality attribute will be higher. (As you can see from a system like the old mainframes!)

[Spooky23]

The advantage that the original cobol people had is that they had functional manual processes to model.

Today, the payroll supervisor has no idea how many processes work.

[giantg2]

Mainframes are expensive because they're niche and basically a duopoly. The companies supplying them know their days are numbered and charge high prices before the market goes dry.

In my experience, the quick teams only learn the surface content. A lack of depth tends to result in more errors down the road and poor architecture. But I guess it doesn't matter since half your team will jump to the next project company in 2-3 years anyways and take that knowledge with them.

[satvikpendem]

> using today's (quality: average B2B startup) programmers and today's methodologies that will write software for 40 years.

I'm not sure I would trust these people over those of the past, regardless of what they say they will accomplish, Chesterton's Fence, and the Lindy Effect, and all.

[johnnyanmac]

I imagine that is exactly what:

>spend more money on business analysis up front.

Entails. But sure, if the government doesn't want to pay for that quality, they can spend just as much or more resurrecting Cobol devs in cryo-stasis in 2099

[jrockway]

I don't think they have to keep COBOL devs in cryostasis. Some sufficiently motivated individual could likely learn it today. There are even programmers alive right now that would see a large legacy system and not immediately suggest a rewrite on their first day. It's rare but it can happen!

[dustingetz]

ha! you wouldn’t even be permitted to bid the project!

[quickthrower2]

The trick is to use JS. (Not anything that compiles to it)

It will be guaranteed to run in 100 years time.

Any other language can die or have incompatibility issues. Over that timespan.

[kube-system]

I can't tell if I'm being Poe's Law'd. JS engines are a moving target and have been ever since it was called LiveScript.

[satvikpendem]

Just use LISP!

[ebiester]

You could use Java or Python and have the same guarantee.

I am not convinced Node will be here in 100 years.

[ak39]

Node might not, but Javascript (even on the server) will mostly be.*

* assuming that the 100 years hyperbole is accepted

[trealira]

Programming languages haven't even been around for 100 years.

There's no guarantee we'll even be running programs written in C in 2123, let alone ones written in JavaScript. 100 years from now is a long time.

[satvikpendem]

> It will be guaranteed to run in 100 years time.

This is never guaranteed. How do you know that nothing will change in JS land in 10 years, much less 100? You don't, that's how.

[klyrs]

Brainfuck is the only language we need. It's feature complete and practically guaranteed to go unchanged in the next millenium.

[monocasa]

Also throwing out there that COBOL was a language designed by the DOD specifically for managing it's bureaucracy.

It was a naval research project headed by Admiral Grace Hopper to program with words rather than formulas like earlier "high-level" "autocoders", enabling secretaries to transfer their business logic into computers. It actually intentionally has a lot of the same structure as a recipe, hoping that this would allow easier uptake by secretaries.

[wredue]

>Of course, devil's advocate would point out COBOL in 60s was written with a lot more focus on lasting in general. They didn't largely have the same general attitude of 'we iterate quickly with our script kiddies' as is prevalent today.

I worked on enterprise cobol for nearly 20 years and can generally attest that cobol code from the 60s is absolutely nothing at all what you’re describing.

It’s brittle and constantly breaks, requiring 24/7 on call support for people to deal with issues so it doesn’t hold up the next days work.

It was not written with any semblance of engineering prowess. Most cobol written in the 60s was written by business managers, not programmers.

The reason it stays running isn’t because it’s battle hardened. It’s because nobody in their right mind would ever pay to untangle the unfettered mess.

[SteveNuts]

It'd be battle hardened to 1960s standards, payroll has changed a _lot_ since then. And if they can't accurately describe exactly how certain calculations are made (or have to do some of those manually outside of the program after it runs), that would be a huge problem in an audit.

This is why big companies use Oracle or SAP for their ERP, auditors are very familiar with it and the way it functions is well-known and understood.

[wmf]

COBOL in 60s was written with a lot more focus on lasting in general

This is the same code that uses two-digit years and doesn't support lowercase letters?

[cwbriscoe]

COBOL ONLY USES 2 CHARACTER YEARS IF THAT IS THE WAY IT WAS CODED. PEOPLE DIDN'T THINK THEIR 60'S - 90'S CODE WOULD STILL BE RUNNING IN 2000. COBOL DOES SUPPORT lowercase LETTERS. GET WITH THE TIMES!

[trealira]

> PEOPLE DIDN'T THINK THEIR 60'S - 90'S CODE WOULD STILL BE RUNNING IN 2000.

That serves as evidence against this point, though:

> COBOL in 60s was written with a lot more focus on lasting in general.

[cwbriscoe]

Yeah, that point is wrong in my opinion. I don't think programmers back then tried to build things to last any more than programmers today do.

[bastawhiz]

> you can't say the code base isn't battle-hardened

Your statement is undermined by the fact that the system has failed its sixth consecutive audit. If it's not hardened against failing an audit, I'm not really sure what it could possibly be hardened against.

[bumby]

Part of Murphy's Laws of Combat: No combat ready unit ever passed inspection. No inspection ready unit ever passed combat.

Managers need to be careful of Goodhart's law. (in this case "passing inspection" may not be the best target)

[bastawhiz]

And yet hundreds of thousands of companies pass the same audits. The call for audits came from ridiculous amounts of misreporting and waste. Accounting systems that can't do accounting and deal with trillions of taxpayer dollars should not have the goalposts moved. A 0.001% error in a trillion dollar system is ten million dollars. That's taxpayer money that could fix derelict bridges or pay for thousands of children to receive lunch at school.

[bumby]

I agree with you, but I’d venture to guess most of those companies would use “combat” in that sense as an analogy.

The DoD does not.

[herewulf]

Yet the DoD's bureaucracy is neither combat nor inspection ready.

[psunavy03]

Having worked on a team responsible for maintaining COBOL code, if from the 80s and not the 60s, it may "last," but it's also a rats nest of dependencies that's a pain in the ass to test, and part of the problem of moving away from it is 40 years of undocumented business logic baked into it over the years.

[giantg2]

Even when I started 12 years ago in Java/JSF/Oracle stack there seemed to be way more focus on durability and longevity. The system I started on was architected in a way that made it easy to modify and integrate with other system. It worked well. Eventually it's scope creeped, which would necessitate modernization. But it was very maintainable and lasted years.

Now the systems I work on are hot garbage because the tech is constantly needing upgraded versions, changes to the code are onerous because of the way the system was designed, we are constantly having data issues for various reasons, and people jump teams every 2 years so there's no continuity.

I'm convinced it's a leadership issue. Push for speed and you'll get it. It will cost you in other areas.

[jrockway]

This just sounds like survivorship bias to me. I am sure there is a lot of junk COBOL that killed the companies that relied on it. Nobody cares today because the business is gone.

Something that's interesting to think about is that because the cost of changing COBOL is so high now, only the most important change requests actually get implemented. Compare that to a team that can develop and ship a change the same day; their efficiency leads to more requests that end up being bad ideas. The Pentagon has designed a system to ensure that only the most well-thought-out ideas make it into production, they just went about it in a very strange way.

[Spooky23]

If my experience with an unemployment system is a guide, the COBOL is fine. The scaffolding around it, different story.

[bogota]

Start paying 250k a year for cobol developers and the problem fixes itself really fast. That or start migrating the system. Or more likely they keep using it until it completely breaks and then run to the government for a big handout to fix it.

Easy to be irresponsible with other peoples money.

[thelastgallon]

Incentives only work on low-skilled labor that is quantifiable. Eg. more money for flipping more burgers. It is not a settled science paying people a lot more will automagically solve problems. People crave Autonomy, Mastery & Purpose: https://www.youtube.com/watch?v=rbR2V1UeB_A

What is definitely going to happen is you will attract the cowboys who have convinced themselves and can convince you to pay top dollar. They will come up with lets do kubernetes + service-mesh + multi-cloud hybrid + blockchain + crypto + AI/MI on this COBOL ... They will take top dollar, make a bigger pile of mess, sell it as a success story and move on make bigger messes.

[VoodooJuJu]

>People crave Autonomy, Mastery & Purpose

I'm not a sack of chemicals whose desires can be reduced down to some pithy pseudo-scientific Maslowian quip. The only thing I crave from my job is cash so I can take care of me and my family. 250k/yr for a COBOL position would suit that end just fine. You can keep your autonomy, mastery, purpose, and all the other foofoo, just hand me the check so I can be on my way.

[wolverine876]

> I'm not a sack of chemicals whose desires can be reduced down to some pithy pseudo-scientific Maslowian quip. The only thing I crave from my job is cash

I thought you were against reductionism?

[VoodooJuJu]

https://www.dictionary.com/browse/reductionism

[Cyphase]

They were replying to someone who was saying something about "people"; they only said something about "I" (themselves).

[johnnyanmac]

>The only thing I crave from my job is cash so I can take care of me and my family

sounds like purpose, of which you need mastery of some skillset to earn your autonomy to furfill.

That said, I'm sure there's much easier ways to earn 250k than untangling a governement codebase. Not that they are offering that to begin with.

>just hand me the check so I can be on my way.

Yeah, you probably wouldn't pass clearance with that attitude if we're being frank.

[whatshisface]

250k is not top dollar, it is average for experienced engineers.

[metabagel]

Outside the Bay Area, $250K is top dollar.

[cebert]

Not really, even in the Midwest, such as the Metro Detroit region, 250k is definitely attainable. I’m in that band and live there.

[myko]

Base or with bonuses?

I find it hard to get roles in the midwest (large metro in Ohio) clearing 200k, though sometimes bonuses get you up there.

[omginternets]

It definitely is not.

[jongjong]

WTF. I worked on some high exposure projects as a senior dev and never a final offer above $130k USD. Even after haggling hard at a company I had been at for 3 years and they told me I was their top, highest paid engineer and it led me to quit. Then I got approached by Facebook to work in London HQ and the offer they were dangling was £100GBP which was about $130k USD. Highest offer I ever got before I even got to the final round of interviews was $160k USD from a well known crypto company but for some reason the head of engineering didn't show up to the final interview at the last minute after I flew through all the tech tests as those devs who interviewed admitted. It was weird.

I always assumed the reported 250k plus salaries were fake news or only reserved for children of powerful people.

[Kranar]

250k is not top dollar for programmers, but it is top dollar in absolute terms. The point is that you don't incentivize better work by paying top dollar. Taking a mediocre developer and paying them more doesn't make them a better developer or even motivate them to become better.

It can attract people who are already good software developers absolutely, but the second point being made is that you also attract a lot of charlatans as well and so just throwing more money at the problem isn't exactly a great solution.

You need a solid culture to go with the money.

[ramilefu]

Depends on where you live. An “experienced engineer” is going to make a lot less if they live in Mississippi than if they were in California. 250k for the former is likely “top dollar”.

[bluefirebrand]

People on HN are convinced if you don't live in California maximizing your salary, you're not capable of doing so.

Highlighted in the sibling comment: "you will get whatever candidates decided to live in Mississippi"

As if top talent never decides to live in Mississippi and accept 250k/year as their salary ceiling...

[1323portloo]

I didn't say that top talent doesn't live in Mississippi, I just said you will get what candidates DO decide to live in Mississippi. If you want top talent, you have to advertise nationally, and be willing to pay the national rate. I live in a city that is quite poor for an employer that is quite large. They pay slightly better than average for the area except for their engineers and upper management, that goes to market rate because they are willing to hire the best available.

[johnnyanmac]

People also seem to forget that there are "cost of living adjustments" for a reason. Many companies will fight and suggest that you don't need salary X if you don't live in place Y.

It's not impossible and $250k is nowhere near the ceiling. But you're not going to find those jobs on a causal LinkedIn search like you would in California.

[1323portloo]

If you are a small shop, sure. If you are a major employer you will either meet market rate and get top candidates (willing to live in your area) or you will get whatever candidates decided to live in Mississippi.

[rybosworld]

This is a lie that is commonly spread here and on reddit.

[bogota]

250k with government perks and retirement is pretty nice. Also for 250k cash you will find lots of people willing to learn cobal.

[Miner49er]

No, that is not average, maybe in high COL areas, but that obviously would then not be average.

[rumdz]

Outside of CA?

[metabagel]

Even in Southern California, $250K is a very high salary for developers, at least in embedded systems, which is the space I’m familiar with.

[winrid]

embedded system engineers have always had low salaries... I don't think I've ever seen an embedded C or C++ job pay more than $130k. I made more than that writing JS almost ten years ago..

[curiousllama]

Cobol cowboys are a thing. They make substantially more than 250k. They’re competing with Wall St though, so hard to just pay more.

Fed govt migrations are ALWAYS shitshows. The army alone has like 20 individual email services. The pentagon is to org complexity what big tech is to technical complexity. It’s turtles all the way down.

This is literally one of the largest bureaucratic challenges on earth. There’s no simple fixes

[mschuster91]

The solution is simple: you need someone with absolute authority over everyone including the generals to lead the effort.

There's a time and place for democracy - but large IT projects are not. Do a thorough need analysis, compare with what's reasonably possible using COTS software, and adapt or discard what is not possible.

[vajrabum]

That's exactly why DoD kept bringing Grace Hopper out of retirement first in 1967. They eventually gave her a star in the 80s, and along the way she was as responsible as anyone for the standardization efforts for COBOL and Fortran including the design of standardized compiler validation suites. Testing and interoperability before testing or interoperability were a thing! One of the later major projects she worked on was modernizing the Navy payroll system. Even junior sailors at the time knew that she and her team had made sure they got their paychecks. She retired from the Navy at age 79 in 1986. Nobody messed with Grace Hopper.

[neurotech1]

Adm. Grace Hopper wasn't the only senior officer who was recalled because of their expertise. One is a retired JAG lawyer (a friend actually) who continues to advise at the Pentagon on several areas of expertise. She actually learned COBOL from Adm. Hopper herself, and always had high amount of respect for Adm. Hopper and how she disliked bureaucracy. There is a lesson there.

Ironically, my friend has programmed COBOL on DoD mainframes, and still does as a consultant, having interacted with them for decades. Back in the 1970s she had a surplus IBM minicomputer that her father brought home for business use, and learned COBOL on that, and used the computer for furthering her legal research.

[count]

I can't tell if you're being sarcastic, or earnest.

This is the funniest possible response. The 'thorough need analysis' is the reason we are where we are now!

The thoroughness is so thorough that the needs change faster than the analysis can be completed, obviating the analysis in the first place.

[nickpeterson]

Part of the problem is, you tend to need cobol experts to actually have any hope of successfully transitioning to another technology, and if we had enough cobol experts would look at the cost of migrating and say, “who needs to, we have a bunch of cobol developers?”

[adamc]

Hire experts and have it be reported as "waste" and hear complaints in Congress about government waste.

We want government to be cheap. You get what you pay for.

[wubrr]

That could work if the existing code is actually up-to-date, correct, and if you have knowledgeable users of the systems backed by this code. The part about 'code corruption' makes me thing this is not the case.

Worked at a big bank migrating some old code to new stack one time:

  - Original developers are gone.
  - Existing users don't understand the systems outside of their own narrow use-cases.
  - Code which was committed to repositories didn't match the actual running code, because people would go into production servers, manually edit the code there, in place, not document or commit the changes to repo. There were like 20 years of adhoc changes made in this way.

We had some business analysts which were supposed to talk to the users and gather requirements - they very quickly simply gave up and asked the developers to 'reverse engineer' everything.

Anyway, it was a hilarious shit-show, but kinda fun at the same time (if you're able to ignore incompetent leadership pressure - a skill I mastered during this time).

[obviouslynotme]

>Code which was committed to repositories didn't match the actual running code

I remember the first time I saw this at a small company. The head programmer was the only one with the actual production source, despite a working source control system. He complained about how people didn't appreciate how hard it is to make builds work. I should have left when we lost three man months of code to losing email. I only left after screaming at the head programmer for refusing to stop making the repository unbuildable at all.

[MrMetlHed]

In the last part of the series from 2013 we have a chart that shows the Pentagon had spent over $10b modernizing their accounting systems (that was through 2013, no idea how much that number has grown.) I can't imagine anyone would miss a few million of that going to COBOL developers. The money scale for Pentagon projects is mind-boggling to me.

[lancepioch]

Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.

[Jtsummers]

Yes, but not for contractors. Most software sustainment ultimately ends up in the hands of contractors with government supervision in the form of program offices. Not all, though, many programs are also "organic", primarily or fully staffed by government employees and managed by government employees.

[randmeerkat]

> Yes, but not for contractors.

Sure, but it’s not like the consulting firms are paying their “contractors” more, they just siphon up the difference for their shareholders.

[ska]

That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates, so long as they get to keep their overhead % fixed, and that's probably how the contract is structured. Win-win (and perhaps -lose for taxpayers).

[randmeerkat]

> That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates

That’s exactly how it works, look at contract government salaries compared to anything in the private sector. They charge the government more as rates “go up”, but that certainly isn’t passed along. If large contracting companies really offered value to the government _and_ kept up with market rates for their employees, the state of federal software wouldn’t be what it is today.

[toasted-subs]

Starting pay for cobol in finance was around 350k last time I glanced. Nobody wants to rewrite a working system which will likely be much more expensive than finding one person to maintain the project.

[saiya-jin]

With rewrite comes uncertainty and risk too, how many new bugs compared to code that works for decades?

[ska]

> Start paying 250k a year for cobol developers and the problem fixes itself really fast.

Do you think they aren't? I don't know any now, but the last one I did was making more than that 15 years ago.

Old crufty legacy systems in a big bureaucracy are many peoples idea of hell. Empirically speaking, good pay and job security aren't enough to have people knocking down your door for that kind of work if they have other decent options.

[ravenstine]

This is a problem not just in software. Today, everyone wants a senior <insert job role> for the price of a junior <insert job role>. So many problems would be solved orders of magnitude more soon if people were properly compensated. Instead, most things in the United States are running on economic fumes.

[encoderer]

I can imagine a future where new software is, mainly, written by humans while legacy software is maintained by ai.

It can take months for a new developer to understand a legacy code base but an LLM with a big enough context window would be instantly productive.

[nradov]

The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case. No LLM can do that.

Once you have the detailed specifications the coding is relatively easy. Some of that can be partially automated using AI CASE tools, but that only gives a marginal improvement to overall project productivity.

[dragonwriter]

> The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case. No LLM can do that.

One of my first one afternoon (for prompt refinement and trying it out) toy GPT-3.5 “applications”—really just a prompt—was having it interview for reauirements and draft and progressively refine a requirements doc in a specified (by reference to an author in popular Agile literature, not by specific templates, so just relying on the model’s general training) format. Its pretty much what convinced me of the broad utility of the technology.

I absolutely think that with a bit of effort, GPL-4-128K, a custom RAG framework and/or the Assistants API, you could build something that handles a lot of this. I’m not really bullish near term on LLMs as full replacements, but I can see it as a big force multiplier.

[DougBTX]

> The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case.

So lots of tedious repetitive manipulation of language to extract facts and transform them into a systematic format? My prediction: LLMs are going to eat that for breakfast.

(Honestly it is a great business idea for someone with the right contacts. Lots of big customers with deep pockets and large private datasets to train against that might now have a solution to a problem which was previously intractable in scale.)

[nradov]

None of those customers have actual data sets to train against when it comes to ERP and accounting software requirements analysis. The data that exists at all is scattered across random Word documents, wiki pages, paper notebooks, and legacy requirements management systems. Most of it is summarized. It's not recordings of past interviews with SMEs that business analysts and product owners conducted to extract functional requirements. In order to train an LLM to conduct such interviews you would have to obtain actual transcripts of such interviews. Not impossible, but unlikely to happen anytime soon.

[charlie0]

Ding, ding, ding, we have a winner. This. I see issues at my small startup with not having clarity with certain billing items. This issue is not technical, it's the unknown requirements of why things were written in the way they current exist. I can't imagine what kind of bizatine rules they have in the Pentagon that were placed there decades ago, and no one can answer why. Codebase must be littered with Chesterton fences.

[Yahivin]

I can imagine the opposite.

[encoderer]

Cool tell me more.

[SteveNuts]

Not OP but I could see some shops pushing AI generated code to production, then when changes need to be made, they can't get the AI to modify the existing code in just the way they need, so a human has to intervene.

[bastawhiz]

I can't get Copilot to generate Python that adds numbers together correctly sometimes. Getting an LLM to generate correct, working code for a language that hardly anybody writes anymore is almost assuredly going to lead to failure.

[encoderer]

yeah I agree but when you look at the slope not the y-intercept it’s getting obviously better.

one advantage the government would have is training/fine-tuning on a hundred million lines of domain specific cobol.

[bastawhiz]

The slope doesn't really matter, because the target is "better than a human, and able to identify and fix its own errors". The slope will decrease as you approach this threshold.

It's also wildly bad to plan to train and fine tune on code that you know has bugs. Already we have Copilot generating code with trivial vulnerabilities because that's what it's trained on.

[TurkishPoptart]

Is the problem really related to COBOL, or is the fact that this institution is basically unaccountable to taxpayers?

[samus]

If it ain't broke, don't fix it, they say. But if it is already broken in significant ways, there is little point in not replacing it.

[chatmasta]

See also, "John Stewart Questions Defense Deputy Secretary on Budget:" https://youtube.com/watch?v=50MusF365U0

[wslh]

I would love to perform static security code analysis to that code.

[xeromal]

Not all the RAM on earth would be enough to process that shit

[tandr]

Are there any SA tools, that would understand COBOL?

[Jtsummers]

https://www.sonarsource.com/knowledge/languages/cobol/

Never used it (and hopefully never will), but yes.