[LaserDiscMan]

I once worked on an in-house ERP system which had been developed over about 15 years by various developers. It was the engine of the entire company, everything passed through it. The CFO and some senior leadership erroneously blamed it for their shortcomings/used it as a scapegoat. When new management took charge, an initiative was started to replace the system with an industry standard solution. Both myself and the CTO (my boss) made it clear that we strongly felt this would not only go way over budget, but ultimately fail as a project.

Having no understanding as to the technicalities involved, the project was given the go ahead by the directors after several meetings with a vendor. After the CTO and I expressed our concerns about the scale of the project and the sheer amount of functionality involved, the vendor gleefully assured us that they were experienced with "migrations of this scale" and were more than prepared, which was music to the ears of the CFO.

Daily 2-3 hour meetings followed (for many months) to define the scope of the project. Within each meeting I sort of zoned out because it became very obvious that no only did the vendor not understand the scale of the work involved, but had started cutting corners everywhere/leaving out crucial functionality, and this was just the scoping stage, no development had even started yet.

I eventually departed the company but kept in contact with the CTO and learned that after 5 years (project was scoped for 2), the migration was abandoned costing multiple millions of dollars with nothing to show for it.

[notatoad]

>The CFO and some senior leadership erroneously blamed it for their shortcomings/used it as a scapegoat

the key thing to understand about ERP systems is that this is their primary purpose. anything else they claim to do is secondary.

[ZiiS]

Which is why they are vastly better whilst in the middle of a tricky migration, especially if it is going badly, and especially especially if it is being done by the most expensive contractors.

[notatoad]

yes, the key is to always have some sort of migration happening so every problem can be addressed by saying "this will be resolved after the new ERP is rolled out"

[datadrivenangel]

And you can decline to do literally any piece of work because you need to focus on the migration.

Perfect scheme.

[8b16380d]

Or “it’s best to wait till the migration complete before our team needs to do anything”

[highwaylights]

The latter half of your sentence suggests this project may actually have been a roaring success (for the vendor).

[paulryanrogers]

Depends on the vendor's goals. If it's to fatten and butcher a pig, and never see any future work from them, sure. If the vendor is hoping to have a sustained profitable relationship then not so much.

[anonymouse008]

Change this from ERP to POS and you've scripted 9/10ths of the pains for many quick service restaurant chains.

Story time: people are dense as hell. This one idiot put in his personal SSN to open $XX/m a year POS account. This account currently manages just over a hundred locations, and this fool's TIN has just been raking in millions for a few years all because 'I needed to get us signed up!' And what's better!? They still can't get out of the wet paper bag, because otherwise "we'll someone might start selling candy or something unapproved" > we'll isn't that's what your franchise liaisons and agreements are for?! I'm just beside myself.

I'm so hyped on this because this one 'immovable problem' prevents me from having API access to my locations accounts. Everything just hurts. /rant

[bombcar]

I swear the only way to get an ERP or POS change to work is to start a competitor using the new solution and wait for them to acquire you.

And then live forever with two systems running side by side.

[invalidlogin]

Would you mind breaking down the situation in a little more detail? Not my field and I find your story interesting.

[cutemonster]

I also don't totally understand, what's a TIN for example (Taxpayer Identification Number?) and why is it making money? (Don't taxes go to the state?)

[curious_curios]

Yeah, it’s the identifier the government uses to attribute income for, and hence calculate the resulting taxes.

In this case it sounds like a midsize business is using some random guy’s number, so the tax authorities are under the impression there’s a guy pulling in millions personally, and a company with no income. Given that individuals and companies get taxed differently in many different jurisdictions concurrently this will be a mess to untangle.

[cutemonster]

Wow that's amazing, thanks for explaining

[89vision]

> the vendor gleefully assured us that they were experienced with "migrations of this scale"

Sales is mostly just lying to collect a commission check.

[whstl]

Quite the opposite, they are experienced with those endless migrations.

French poet Paul Valery once said: "A poem is never finished, only abandoned".

It is the same for those projects. You just gotta keep paying forever.

[lcnPylGDnU4H9OF]

It's inevitable that Sales will put Product in a bad spot because they're too good at selling their own innocence.

[talldatethrow]

I was a top 0.1% salesperson nationwide in car sales. I read a bunch of lean startup stuff, learned to make a minimum viable product, and then started selling to companies with 20-80 employees. I'm likable and good at selling, so I got 34 companies using this garbage I made. It's the worst nightmare ever to keep people motivated to use it, keep fixing things I made as a rookie developer, keep adding or saying no to features.

Overall the product isn't really needed, and sorta sucks too. If I was an typical developer trying to pitch a startup idea to businesses, it probably would have never got off the ground and nobody would have wasted any time. Maybe eventually the developer would have landed on an idea so good it had REAL PMF, and made that.

But no, instead I sold some garbage and now I'm stuck working on it. There is such a thing as being too good at sales. You don't want sales talking people into bad ideas.

[tomcam]

Amazing story. Love your honesty.

Is there a world where you hire someone to build a more maintainable replacement?

[talldatethrow]

It really wouldn't solve the main problem which is that the product itself isn't really needed or that great, code quality aside.

[tomcam]

But it seems like 34 companies would feel differently?

[jononor]

Hehe, reap what you sow :) But - you are not really stuck though. If you have managed to go from 0 to 34 customers before you can do it again. It is totally an option to drop that product and/or startup, and go with a better one (guided by your learnings). Up to you to decide what is the best way forward :)

[morpheos137]

This is hilarious. Can you describe the product or what it does? Is it something to do with payroll, benefits, or HR?

[talldatethrow]

Construction project management for a certain segment.

[randomdata]

> now I'm stuck working on it.

Why not put those sales skills to work and sell the business to someone else?

[talldatethrow]

Because there's a difference between selling a rough car you detail inside and out and sell for a good deal, versus selling a lemon.

Deep down I know the product isn't that well liked, and a lot of the renewals are based on my salesmanship. I know I have a chance at fixing the bugs, because I wrote everything, while someone else would be screwed IMO. It just wouldn't be fair to sell it to someone because I know they'd be making a mistake.

[codingdave]

It sounds like you are underestimating pretty much this entire industry. Picking up codebases that are sketchy and new to us is just part of the job. Finding and fixing bugs is an exercise to get rolling, not an insurmountable challenge.

When people keep using software, it is because it solves a problem. They might not like the software but begrudgingly use it anyway because it solves that problem. So sure, pat yourself on the back for good salesmanship, but also realize that they keep using it because even if it is covered in warts, it still does the job.

[throwaway14356]

imagine a software buying consultant. You could sell the service to your existing clients.

Long ago i made a few html websites for businesses then explained basic html and ftp to the owner. It was perfect except from other webdesign shops hammering my clients.

[epr]

> Sales is mostly just lying to collect a commission check.

Depends on the type of sales. A pretty good indicator is how many times a customer makes a purchase from the same salesperson. If it's just one purchase (like ERP consulting) it could be grifty/etc like in this case. If they're buying from the same guy for years, there's often very different types of salespeople. I used to think all salespeople fit the sketchy used car salesman type, but after working with great salespeople I know better. This is a big blind spot for us techical/engineering types.

[raincom]

When you are working with the same sales people multiple times, it is an instance of treated prisoners dilemma. Used car salesman is just one time prisoners dilemma.

[taneq]

They’re not even lying, usually. They just don’t have the expertise to tell the customer exactly what they can’t have. That’s why you have sales people.

[mamoulian]

It infuriates me that all of these aren't met with a 'if it turns out you can't, you need to make it right at your cost' clause.

Estimation is hard but it seems it's far too easy for the vendor to blatantly lie with all the risk on the customer.

[rm445]

That is surprising, the industry of implementing industry-standard ERP systems is pretty established. I'd have expected it to go way over time and budget (they always do) but eventually succeed. Usually success is in proportion to the company's willingness to admit they're not special and can do the same things as other companies instead of having everything bespoke.

[calvinmorrison]

I manage a team of Consultants at a small ERP firm focused on mostly manufacturing and distribution

the #1 cause of failure is summed up greatly by Isaiah Bollinger, paraphrasing "most bad implementations are because people are trying to buck the system they bought, rather than work with it, understand how your ERP, eCommerce or other system does a workflow and match it. There's billions of dollars going in and out of Shopify (or x system) daily, and you are not that special. You will spend 10x as much trying to NOT use the system rather than trying to use it".

[swatcoder]

But they don't try to "buck the system" just for the sake of it.

Overcoming process inertia is profoundly expensive and often demoralizing to teams. The project budget for a new system is often pitched as vendor price plus some internal oversight, but this fails to represent the cost of the project exactly because adapting the workflow of a whole division or organization inevitavly costs some multiple of that budget while vendors, consultants, and internal spearheaders all pretend it's negligible.

You're right, ultimately, that failure to adapt is the final damning issue in many of these projects but the root cause of the failure is often that nobody sincerely quantified just how costly and disruptive it will be.

[calvinmorrison]

I'm in small and medium business. A lot of homegrown and unique processes that are just dandy. I mean it they're fine. But when faced by things like standard GAAP accounting processes, or even EDI. You can't argue with it. You can't tell walmart oh no well actually I didn't mean to send that 850. It costs you money. You can't willy nilly charge credit cards anymore. Its just life. It makes it hard for them not just in IT but also to hire new people or replace retiring ones. Only Bob knew this process.

What makes the clients we work with great and unique and what I love is the products they make or the problems they solve for customers, but they're not tech companies or banks. The ones who succeed are the ones who focus on thier core value and core skills and not random accounting process X Y or Z.

[wakeforce]

I work specifically with small and medium businesses too, and can echo that sentiment! I'd love to know more about what you do. Love working with SMBs but don't know a lot of other people in that space.

[calvinmorrison]

Its nice. It's certainly not high faluting as a lot of HN jobs but I make good money for where I live, our clients are mostly regional but a lot of the 50 states have customers over the years. Mostly we work on implementing and servicing ERP systems. Our differentiator is our skillet in integrations and holistic problem solving. Since the ERP sits in the middle of almost every IT venn diagram, we run into new tech on a weekly basis.

Besides the ERP consulting bit we sell niche solutions in our product space. Mostly comms(EDI, ecommerce,etc) or payroll/bookkeeping addons.

[_DeadFred_]

Especially when the processes are beholden to certification authorities (say the FAA, FDA, etc) because the processes has to be approved and cert'd from the agencies. That's a fun one to unravel and co-ordinate getting changed in a timeframe keeping higher managements super optimistic timeframe that they decided to commit to, say the board, on.

[AllegedAlec]

> most bad implementations are because people are trying to buck the system they bought, rather than work with it, understand how your ERP, eCommerce or other system does a workflow and match it.

This is insane cope. We make technology to assist end users perform the tasks they do. To say "well you're doing the task wrong, the tool is made so you do task X way instead" is to put the cart in front of the horse.

[whstl]

It's not really anyone saying "you're doing the task wrong", it's more like "you're doing it only so slightly differently that we need a few months to write customizations for it. We'll send you the bill".

Those big ERPs are not really blank canvases, but when you need those micro-customizations in the wrong place, they can become one.

End users involved in the integration don't really want to learn new processes or even do things slightly different, as they know that changing process often involves burning a lot of political capital and they often lack awareness to know that "just using the ERP the way it's intended" is cheaper. And consultants are experts in finding a chance to perform those micro changes. It's a perfect marriage.

[J_cst]

I understand your point, but that's not how things work. If you source the right ERP you will be presented with procedures and processes that have been streamlined and optimized leveraging previous experiences of literally hundreds of businesses. Smaller and larger than yours. It's an error to dismiss the standard solution without deep consideration. Any process into a good ERP (e.g. managing a deposit payment, managing stock, documents transformation, uniqueness of product codes, and so on) is battle tested and may be ready to solve issues that at the moment your company in not seeing, but perhaps will have to face with the growth of the next five years and in five years perhaps you'll see the reasons why things were setup that way into the ERP. I have been happily humbled more than once by that. Check how things are supposed to work into your ERP, understand them and comply. That's how to have a functioning ERP in your company. Or just don't buy it and go for excel, it's cheaper (but just at the beginning, be warned).

[calvinmorrison]

Just to add... a lot of processes are very defined. I know HN loves to startups that reinvent the wheel but things like GAAP are pretty much the same workflows in any biz. AP, AR, SO, POs, etc.

[calvinmorrison]

A cope with what? Thats literally correct. The tool was made to replace 5 assistants, accountants and paper pushers. There's a design to this ginormous system that works best when you lean into it.

Its why picking the right ERP is very iimportat

[_DeadFred_]

Not really though. We make technology to make SYSTEMs/Processes work, not individual tasks. Take ERP to an MRP level (manufacturing resource planning). It doesn't matter how it works best for Bob, we need to be able to have the part Bob schedules/orders from vendors/makes/QAs/builds from lower level parts feed what they feed when needed in a way that we can consistently/correctly plan, and that follows possible certification authorities requirements/pre-approved processes. This gets especially complicated on say an aerospace production line that has in the high five figures of different parts, with complex individual build configurations with each component in configurable items having it's own lead time, and where everything is JIT because otherwise you would have to sit on inventory levels worth multiples of what the entire company is worth, and every individual item/lot/etc needs to be tracked in perpetuity.

[jiggawatts]

I've noticed that the same applies to any large inflexible platform, such as the public clouds.

If you do things the "native" way in Azure or AWS, you'll be fine, just like millions of other customers.

If you try and make the cloud work like your old data centre platform, then you'll have a bad time.

I just watched a customer spend $2M to deploy software routers to replace the "bad" cloud-native routers. Now everything is more difficult, slower, and just all-round bad. But they "had" to do it. (Narrator: No, they didn't.)

[thayne]

The problem is it is often very expensive to adapt other components to fit the inflexible platform.

In your example, the customer may have had software that depended on the routers having some functionality that the cloud native routers didn't. Sure if they had designed for that cloud from the beginning it wouldn't be a big deal. But now, that $2M might be less than the cost of changing all their other systems to work around the limitations of the cloud native router. I've seen situations play out like that a few times.

[jiggawatts]

That kind of logic seems to make sense at first, but just confirms my point: trying to change IPv4 to suit you is a fools errand. Change what you do to hit ordinary bog-standard IPv4 instead and miraculously you’ll have fewer impediments.

[jpc0]

You clearly have no idea how complex routing is if that is your take on this.

[calvinmorrison]

I usually ask, is it a differentiator for our business to run X?

[epolanski]

I feel the same above technology in general.

E.g. many professionals jump from technology to technology, rather than mastering how get stuff done with one so they end up being mediocre their whole career.

[jajko]

A wonderful statement straight out of ERP's sales guys (or other folks on the same side dependent on keeping the cash flowing). FYI what we trash here are typical SAP-level migrations and all horrible stories that always come with it, maybe your tiny company does things better but then its a different story in a different market.

I have yet to meet a single company which works like ERP are designed to work. This is their edge over competition, their reason for existence on brutal market. And ERP wipe that out, with the most expensive wiper you can imagine, while selling various bullshit left and right, and constantly lying to given company how everything is fine and under control.

Truly, a way to kill a company. The fact that some survived it all to tell a story just shows how resilient whole such org is to such a massive stressor that ERP migration always is.

They sell first and foremostly a lie - that you can have cover-it-all system just like competition, to match your unique way of working, without suffering tremendously, fit like a glove out-of-box (when reality is exactly opposite). It just never works, more like hammering a concrete glove on your progressively more disfigured hand, while being told how rosy your future will be.

[enraged_camel]

I worked in consulting for over a decade, and... yeah. When people migrate from one system to another, they try to make the new system work exactly the way the original system does, especially if the original system is homegrown. That lack of flexibility tends to be responsible for more than half the cost (and time span) of the migration.

[datadrivenangel]

"We hate everything about our system, get us a new one"

"We want the new system to work exactly like our old one"

It's probably not worth the money...

[drekipus]

I haven't pieced those together but of course they go hand in hand.

I had one guy want me to recreate Microsoft word, worts and all, because his current Word wasn't doing it for him

[nilamo]

Or refusing to use anything except Office 2003 because newer versions are too different...

[gentschev]

There's also the related failure mode of deciding you're going to fix all your other system problems as part of the migration, bloating the scope and creating too much complexity. Just do the necessary stuff, migrate, and then go back and fix the nice to haves.

[primax]

So, basically throw out everything that made them unique and differentiated, and do things like the Germans do.

[whstl]

Not quite.

I never worked in those ERP companies, but a few times I've been on the receiving end, working at companies undergoing a large migration.

It is very often stuff that doesn't really matter, is highly inefficient, and requires small changes everywhere in the system. Death by thousand papercuts.

There is no incentive from both sides to change: the company wants to keep modifying to get $$$, employees don't want to change how they work (because change is often stressful), and the person paying the bills is not getting the full picture.

If there's anything that is actually really "unique" (in a good way), then you spend money. Often this means not customizing the ERP, but actually writing new software that integrates with it.

[doctor_eval]

Right. I’ve spent weeks creating automations in a business I knew very well, that would most likely save a few minutes per month for a low salary employee.

But the RFP said I had to do it, and the customer insisted that I follow the letter of the RFP, rather than hit the high value parts of the project.

The two problems with this are that the cost of automation far exceeded the possible future cost of the manual process, and working on this automation took oxygen away from the stuff that would help people.

Needless to say, it was a disaster.

[whstl]

That’s a great point. Very often the cost of automating something in those large systems is larger than the cost of a minimum wage person spending a couple hours a day clicking stuff…

[J_cst]

6 years ago I was called as a contractor by the company I work for as they were desperate that their ERP was just a money drain and despite the money was not functioning at all. Fast forward 6 years. I'm a manager at that company, responsible for business processes and it systems. The ERP works great, everybody's happy. Secret recipe: dismantle custom company processes. Culture change: realize that company's not that special and don't need special ERP recipes. Takeaway: if the ERP standard is set up in a certain way, probably there's a very good reason underneath that setup.

[thefourthchime]

Story as old as time. It's a second system, they rarely work. The only hope is to come up with something small and exciting the a new generation moves too. Then you can build it to become the new thing.

Trying to build a total second system is nearly impossible if it's big and old.

[mixedCase]

Damned. I'd be tempted to write down in my resume "Company would've saved N million dollars and 5 years of their time had they listened to me" were it not such a double edged sword.

[nick7376182]

Better to write "managed critical scoping phase of $xx migration". Not your fault the company priorities later changed and it wasn't completed!

[jmole]

No one gets credit for "I told you so" - look at it this way, you didn't have the skill at the time to make the company listen to you.

[joshuahutt]

How could you, in such a case?

[nilamo]

They ignored the CTO on an issue that was 100% tech based. There was no winning, the board decided to ignore the person they hired to make that sort of decision for them.

[jmole]

I don't know, and maybe there was no solution. Just offering a different perspective.

[joshuahutt]

Some things seem inevitable. I've really run myself through the wringer trying to figure out "what could I have done to achieve a better outcome for everyone involved."

I guess success and failure are both group efforts.

[elwell]

short the stock

[blantonl]

Consulting:

If you're not part of the solution, be part of the problem.

[ethbr1]

Or to put it in other terms, the boxer who throws the fight also gets paid very well.

[kgeist]

Wow, we had the exact same situation. We had a 15 year old CRM/ERP developed in-house, full of very specific and complex business processes/rules/entities tailored to how our business works. At some point, someone decided to migrate to Dynamics CRM. As it wasn't realistic to migrate immediately, they chose to do it incrementally: the both CRMs would be active at the same time, and there would be lots of data synchronizations between them, and our managers/the system's users would learn how to use Dynamics CRM in the meantime. The project was scrapped after 1 year because they realized that fully migrating it would took many years, and the managers didn't like the new CRM because it was lacking functionality (didn't match the existing workflows), and overall the whole synchronization stuff was pretty fragile and full of bugs. Now we've chosen the path of improving dev practices in the existing CRM to impove its reliability instead of hoping that somehow magically a new CRM would solve all our problems without any effort.

[TecoAndJix]

Hershey? https://pemeco.com/a-case-study-on-hersheys-erp-implementati...

[client4]

The nice thing about consulting at Hershey is that they have candy closets where you can grab what you like, it occasionally includes new demo candy. And the whole town smells like chocolate.

[thih9]

What happened to the CFO?

[datavirtue]

Hmmm, maybe check his LinkedIn where he brags about the successful transition and ERP migration he led at that company.

[permalac]

That happened in my company. The CIO left bragging about implementing itil on servicenow. What they did was replacing RT for servicenow, and only for some ticketing queues. Nothing else, no change management, no cmdb, only an incident and non standard requests queues.

128k/ year, while RT still runs in a VM for all the other ticketing queues.

Lots of powerpoints.

[frontalier]

probably got hired by the vendor

[dxxth]

I bet it was Oracle, SAP, or IBM. The hydraheaded horror of enterprise bloated licenses.

I feel this on a deep level since the ERP system at our company is in the middle of moving to one of these big vendors and so much is simply not working out, extending timelines and development cycles. Cost only goes up and ROI is pushed out beyond the horizon.

PLM is equally bad and a horror show of legacy vendors trying to sell their solutions with the promise of customizability. Again the small, more modern players in the PLM space are constantly ignored for the big legacy ones and it turns out those legacy platforms even in their latest iterations are inflexible at best or downright hellish at worst. The reason behind all of this? Service contracts and vendor lock-in are the main drivers of value for these vendors, rather than quality (and modern) engineering.

[isakkeyten]

I've been on the other side of this. Big company was under the impression they could ultimately save money by switching from an enterprise ERP solution that costed them hundres of thousands of dollars in licences with an inhouse one. The only problem was the ERP was carrying 90% of the company on it's shoulders and this would've been a horribly large task. I worked on it for 2 years with incompentent managers and slow progress. After I left the company they continued for another 2 and then quit that endevour. Wasting a shitload of money (think 10 devs + managers for 4 years...).

[callwhendone]

>I once worked on an in-house ERP system which had been developed over about 15 years by various developers.

I read this and immediately became sick to my stomach. I despise working in corporate america.

[lostlogin]

> I despise working in corporate america.

America might lead the way, but corporate anything, anywhere is pretty terrible.

[justinclift]

Was SAP or IBM the external vendor? :)

[some_guy_in_ca]

I've spent my entire 30 career in ERP Consulting (PeopleSoft), working with Payroll, Accounting, and Student management systems.

I have a current client that is a mirror of this exactly. They had one new executive go to a sales demo, and the sales guy told him it would be "live" in a few months, and he believed it. It was so laughable and downright embarrassing for the client. In this case, I know it is at least a 3-year project if you have competent team members and vendors. It also depends on how you define "success" and how many business processes you are willing to break. However, these can stretch into 5 year projects when you are working with mission critical systems running accounting and payroll. In those cases, you cannot move fast and break things.

Most organizations don't have competent teams, and the vendors are often made up of low-skilled offshore workers, which makes nuanced, complex projects very complicated. The sales side will promise the world then the implementation team is low skilled and underdelivers.

I have seen this with several other clients, replacing pieces of PeopleSoft modules with different applications. Sometimes they come back and sometimes they just suffer through it and then find a third option after the new system fails to deliver.

Most ERP systems are just CRUD systems. So, you end up with a working CRUD system being replaced by a new CRUD system. The old system had its strengths and weaknesses. The new system will solve some problems and break others.

You spend 3-5 years replacing a system only to end up in the same place you were before: A CRUD system. This is maddening unless you are the new vendor and getting enriched.

[throwaway5959]

I’m sure it worked out well for the pocketbooks of the new management.