[wmat]

I used to work in the insurance systems industry. There are many, many expensive problems that have to be solved before legacy systems can be replaced. In no particular order:

* The Myth of Uniqueness: Each and every company doing business believes that their business is different and couldn't possibly be the same as another company's. This is not completely untrue, but it's also not completely true. The problem is that this myth is propagated by the enterprise software vendors. The profit margins on services to customize software is far, far greater than the margins on software licenses. It is in vendors best interest to promote software customization.

* 50 Years of Customization is Hard to Throw Away: OK, maybe there's not many 50 year old systems running in production any more, but there are 25 year old systems. The amount of money spent to customize those systems over the years is enormous and big corps are not eager to toss that money spent away to start over.

* Database Conversion is Hard: Not so much difficult, but hard in the sense that it requires many, many hours to first clean up the old database, run the conversion, then validate the new data. One mistake could mean that someone's policy lapses because the conversion broke the automated Pre-Authorized Cheque processing code for that one client.

* Working at Big Corps is Uncool: Let's face it, being a programmer at a big bank or insurance company is perceived as boring. The over 40 set that these jobs appeal to aren't traditionally big adopters of new technology. As these people get promoted into decision making roles, they default to safe, known technologies.

* Corporate Partnerships: Big financial industry companies outsource their IT to corporations like IBM. These services companies have a vested interest in maintaining the status quo. It's much like bee-keeping bears ;)

* The old stuff still works: Even though the legacy code is unreadable and numbers in the millions of lines, it still mostly works. The "if it ain't broke, don't fix it" argument is an easy one to make when your executives (beholden to the shareholders) have to make their quarterly numbers.

Don't get me wrong, I'd love to see start-ups in the financial services software industry pop up with innovative solutions. In fact, I've thought about attempting this myself, given my background. But the elephant herd of legacy vendors to find your way around is extremely intimidating. In fact, I actually saw a few small vendors pop up over the years, get small contracts within the big insurance company to do something innovative with their shiny new technology (but not mission critical to the corporation), and then watch that company take on more and more lucrative services contracts further from their core software product. Eventually, their initial product offering becomes a footnote for them and they turn into something else, legacy related.

[agentultra]

http://www.amazon.com/Working-Effectively-Legacy-Michael-Fea...

Working with legacy systems is a black art that I didn't learn about until I took a job supporting and extending one such system. The book I link to above was critical in helping me to understand the approach taken by the team I was working with. It takes a keen, detail-focused mind to do this kind of work.

The approach we took was to create a legacy interface layer. We did this by first wrapping the legacy code within a FFI. We built a test-suite that exercised the legacy application through this interface. Then we built an API on top of the interface and built integration tests that checked all the code paths into the legacy system. Once we had that we were able to build new features on to the system and replace each code path one by one.

Unsurprisingly we actually discovered bugs in the old system this way and were able to correct them. It didn't take long for the stakeholders to stop worrying and trust the team. However there was a lot of debate and argument along the way.

The problem isn't technical. You can simultaneously maintain and extend legacy applications and avoid all of the risks stakeholders are worried about. One could actually improve these systems by doing so. The real problem is political and convincing these stakeholders that you can minimize the risk is a difficult task. It was the hardest part of working on that team -- even when we were demonstrating our results!

The hardest part about working with legacy systems are the huge bureaucracies that sit on top of them.

[wmat]

I've read that book as well, and highly recommend it. It's excellent.

[gutnor]

You forgot

* It is near impossible to measure the savings of doing a conversion.

* Anyway, a huge chunk of the subtle behaviors of the current system is unknown, or when known, the business reason is lost in history. That makes it near impossible to properly estimate the cost a conversion would be.

* All the system in legacy tech are core product - they affect literally everyone in the company - thousands of people in the company. Over 40 that still like new tech have rock solid incentive not to push for them.

[wmat]

Yep, definitely agree with those 3 additions.

I've thought long and hard before about how to supplant a current legacy system that's taken root like an old oak tree in a financial services company. I honestly don't know how.

For example, a Life Insurance Administration System in a small insurance company. First, let's make some broad assumptions to make things easier:

1. The system will be "go-forward" only. i.e. we will not convert current policies onto the new system. 2. The system will NOT support any current product lines, but at launch will only support new product X, which is still in the product planning stages.

Given those 2 assumptions, how much customization will the new system require? The answer that everyone hopes for when they buy the software from the vendor (and the story the vendor sells) is, very little. The reality is that the system that goes live is usually very, very customized OR it ends up being very under utilized and simply interfaces WITH THE LEGACY SYSTEMS to meet the requirements of the product team, the regulatory bodies, the actuaries, etc. etc.

So what's a legacy system hating hacker to do?

The only way I could see legacy replacement being feasible is if big corp X decided that they were going to do full in house system development, and then market the core or the resulting system to their competitors. This isn't unheard of, and has been done in the past, but is pretty rare. The trend over the last 20+ years has been to out source virtually all of IT and buy off-the-shelf, vendor supported software to administer daily business. The argument, that big corp X is not in the business of software development is difficult to argue against. Alas, the end result is vendor lock-in, increasingly expensive services contract, and stagnant systems that are impossible to pry your business loose from.

The in-house/out-source cycle is fairly common in big financial services/health care corporations, usually on about a 4-8 year schedule.

I suspect the HN audience is not particularly interested in legacy systems development, or how one's insurance policy is administered. But, the next time you notice how large your insurance premium payment is, or complain about bank fees, or drug dispensary fees, etc. etc., stop and think for a second about what that money is paying for. In the case of insurance (which is the industry I'm familiar with), I'd argue that for a low-risk client, that a large percentage of your premium goes to administration of some sort.

[tbrownaw]

The only way I could see legacy replacement being feasible is if big corp X decided that they were going to do full in house system development, and then market the core or the resulting system to their competitors.

Or do full in house system development and then buy out their competitors. A while back at one of our quarterly IT meetings we had a chart showing this, X number of competitors "recently" bought out, Y number of claims-management systems (and I think other related systems, things like bill review) migrated to our in-house system and shut down, Z number of systems currently being migrated.

[wmat]

That's definitely another angle I hadn't really considered. So would current vendor clients then purchase support packages from you? Or would you encourage their conversion to your new system?

[mgkimsal]

Often it can be easier to sell 'customization' of software vs 'reworking' a business' internal processes. Internal process change likely requires upgrades to other systems and retraining. "Don't worry about changing XYZ - we'll just customize this software to work exactly like you work now, so you don't need to change anything!". Except.... "exactly like they work now" is probably inefficient and/or antiquated. Upgrading business processes to take advantage of what entirely new systems can offer would probably result in more efficiency and competitive advantages, but it might be several quarters or maybe a year or more before that ROI plays out.

[fiatmoney]

There's also the issue of the vendor having effective control of the legacy system. When IBM is the only one with expertise to actually change your 30-year-old mainframe setup (as opposed to just keeping it running) they have a lot of leverage over any transition process. "Nice database you have there, be a shame if anything happened to it" ends up being the implicit scenario.

[wmat]

That's an excellent point! And one I forgot to make. It's a pretty sweet scam if you're a vendor.

I've been in situations where critical system changes to support new products were being described in Business Requirement documents, sent to the vendor for translation to Technical Requirement documents, returned to the client for approval, and then scheduled for implementation at some point usually 6 months beyond the original requirements process completion.

Could this be why there are NO widely used open source business systems in the financial services, heath care, and banking industries?

[andy_boot]

It's not widely used but the investment bank I used to work at open-sourced some of their work. They figured they could get more eyes on the code and find issues with it faster.

https://www.openadaptor.org/

source: http://www.computerworld.com/s/article/57362/Open_source_bre...

[skue]

Everything you so aptly described also applies to the healthcare industry, which is mired in a language from the 60s (MUMPS).

[sampo]

Here in Finland (population 5 million) the government is preparing to allocate ca. 1500 million euros to order a country-wide medical patient database from Accenture, who are the local representative for Epic Systems [1] and Epic's MUMPS-based [2] medical information systems.

[1] http://en.wikipedia.org/wiki/Epic_Systems

[2] http://thedailywtf.com/Articles/A_Case_of_the_MUMPS.aspx

[eevo]

My apologies. I live in the area where Epic is based. They tend to hire straight out of college and work their employees into the ground. It's a weird subculture over there, but they're swimming in money for all the government kickbacks hospitals get for implementing electronic medical records here in the States. http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRI...

[wmat]

Absolutely. Ironically, the last time I encountered the MUMPS language, I was also dealing with a whack of COBOL systems.

[mechazoidal]

Don't forget about Caché Intersystems, who provide a "modern object-oriented" MUMPS implementation. Complete with SQL emulation and a web template language that's almost a direct lift of ColdFusion(down to needing Dreamweaver to edit files stored inside the DB).

[Roboprog]

I interact with another team almost daily that has to support that. They do their job pretty well, but I really wished we simply had something like MongoDB to store some data in for a recent, rather trivial, project.

It's indeed a complicated, convoluted, growth.

[gaius]

-1 for ageism. What's worse, that, or persuading a client to port their app from RoR to node.js just because it's "cool"?

[wmat]

I was trying to keep ageism out of the discussion. I'd argue that moving to the latest and greatest must serve a real business purpose. That purpose could be to leverage modern languages to build a more extensible, flexible, and maintainable system. Technology for technology's sake is rarely a positive proposal.

[mgkimsal]

Unforunately, it still does creep in.

The 'real business purpose' argument must also take in to account skills of available workers.

If no one is able to be hired to maintain and extend legacy system X, you're left with

* hiring workers with modern skills and backtraining them on older/legacy/custom stuff that won't do them much good in the future market, so they may balk (the smarter ones would demand a hell of a lot more money).

or

* converting to something more modern and perhaps somewhat future proof. The code itself might not be future proof, but the processes the business puts in place during the conversion should take in to account future conversions and upgrades as part of the requirements.

[jjohns]

Very insightful.