[Nexialist]

My worst codebase story:

In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.

In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.

Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).

Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.

This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.

Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.

It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.

[cjbgkagh]

So they had a problem, got 2 years of approved development effort of a small team to solve it property which they did successfully, and then you resigned? After they fixed the problem?

Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.

I presume there were other reasons for resigning, or you just like massive refactoring projects.

[Nexialist]

It was a little tongue in cheek, but yes. I had large grievances with the software culture there, but after I got sign off on the project to modernise our build process, I couldn't bring myself to abandon ship in the middle of trying to fix it.

After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.

[gnat]

Thank you for the clarification. Because you said “it took a small team … and then I resigned”, it was unclear that you were part of that small team and instead made it sound like you left because the problem was fixed.

[dfee]

For what it’s worth, it wasn’t unclear when I read it.

[thanksgiving]

I worked at a company recently where I couldn't get a team to do the fairly simple task of cleaning up our git ignore (so we wouldn't have to painstakingly add certain files to the index every time we changed something) so I take it as a massive accomplishment moving to git within two years.

If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.

[duggan]

Yeah, maybe it's one of those reading between the lines things, but I think anyone familiar with burnout would understand.

There are few efforts as truly thankless in many organizations as fixing things.

[brightball]

Had a similar thing on a small team at a bigger company. We didn't have any type of version control so the team did all development on a shared dev server that we SSH'd into. We'd use vim to edit files one at a time and rely on vim's file locking to make sure that we weren't editing anything at the same time as anybody else.

Oddly, the project was one of the best I've ever worked on.

[mikeocool]

I recall something similar from my first job, except the shared file locking was a full on feature in Macromedia dreamweaver.

CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.

[mnahkies]

Macromedia flash / .fla files weren't particularly ideal for collaboration either, though I still feel a bit nostalgic for working with flash in general

[evan_]

It just did this by creating a sentinel file- so if you needed to, you could just delete the file manually.

[masklinn]

SVN provided the ability to steal locks, and locks were opt-in so e.g. you wouldn't have made CSS need locks because it's a text file and merges fine. Mandatory locking was mostly for binary work files e.g. PSD and the like.

[mikeocool]

Good to know, though I’m optimistic this is knowledge I’ll never need again!

[q7xvh97o2pDhNrh]

...and whoever did that "accidentally" when they had to leave around lunchtime on Friday was, presumably, celebrated as a local legend.

[ufmace]

I think a lot of people who've only learned programming in the last 10 years or so don't realize that Git, and even more so the large-scale popularity of Git, is actually pretty new. A lot of programming happened before Git was created, and before it became mature and popular enough to be the default for everyone.

Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.

Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.

I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.

[0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

[fiddlerwoaroof]

I use CVS, RCS and Subversion on my personal projects for several years before I learned git. I don’t remember any of them being a pain to setup for small projects and Sourceforge provided free CVS hosting for small projects.

[brabel]

I started with Rational Clearcase, which I interacted with via Eclipse, and I remember that you did have to lock a file to make changes to it... pretty sure there was no merge capability (but maybe I was just too junior to know stuff like that, it was definitely not something you would do as a matter of routine, like we do now with git). Someone leaving for vacation and forgetting to unlock a file could be a huge pain in the ass.

After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.

[redserk]

Rational Clearcase is a disaster but you could do merges.

The fun/frustrating part was getting branching and "views" correct.

I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.

[ufmace]

I've never actually set up CVS myself, so I suppose I can't accurately say how easy or not easy it is. The "nobody" might deserve an asterisk - I'm sure it's not literally not a single person, but it seems to be pretty rare in my impression. Nevertheless, AFAIK, the non-distributed nature seems pretty limiting.

In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?

[fiddlerwoaroof]

I don’t think so, it’s been a while, but I think there were tools for copying history between cvs servers. I don’t remember CVS being particularly difficult to manage.

[pbh101]

I set up SVN for myself before Git and IMO Git is definitely easier.

[eastbound]

I’ve used Clearcasse in 2006 and it had all the branches that we needed. We didn’t use locks because that was annoying. It was at Deutsche Bank and we were 200.

When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).

[FartinMowler]

I once worked at a corp that sold software. For the exact same software product, maybe 10-20% of customers received so much benefit we could have charged 2-3x the price and they would still be very happy. On the other end, for maybe 10-30+%, the exact same product ranged from painful to a disaster. Often because they would try to force it to work the way their pre-existing paradigms worked instead of taking the time to learn how it actually worked.

[folmar]

> source control was clunky and a pain in the ass to set up

I concur. Creating a CVS repository was not harder than creating a git repository now `cvs -d $dir init`, and using, typical for the time, shared network drive, gives server experience without setting up a specialized server. It was just not included with Windows/typical Windows dev tools, so you had to know it existed, and knowledge was slower and harder to get.

[stavros]

We all take too much for granted how much git "just works", even when it doesn't.

[Izkata]

Even those before git - what they're describing sounds a bit like RCS, the precursor to CVS, which came before SVN.

I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.

[wolf550e]

VSS6 had file checkout, which IIRC locked the file preventing other devs from editing it. It was popular with Visual C++ 6.0, circa the year 2000.

https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe

[pelagicAustral]

This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...

[DaiPlusPlus]

MSAccess is a tragedy, imo. Right-up until, say, Access 2007, it was a simple (in a good way!) RAD platform following those zombie 4GL predecessors, but it’s been left to stagnate without any real effort to modernise it, and it’s clear from how Microsoft’s been downplaying Access that it’s a product they’d really rather not have to support, but they know if they do kill it then it would weaken on-prem Office’s moat and everyone will probably just rush to Google Firebase.

Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.

[matheusmoreira]

> It took a small team two years to migrate the entire codebase to git with proper CI

I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.

[wredue]

I’ve worked on “check out” code bases very similar to this. I mean. Nothing so insane as scripts patching class files in to jars Willy nilly, but it seems the “just check out a file so nobody else can work on it” thing is a “common” “solution” to this.

I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)

[wiether]

> In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.

> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.

COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!

But we were using Subversion's lock system and had a proper CI/CD with Jenkins.

So you won, your project was worse!

[werdnapk]

Visual SourceSafe would show that a file was checked out hinting to maybe stay away. Good times.

[telgareith]

Did you know, theres a GitHub repo called [you-poor-bastard](https://github.com/SirkleZero/you-poor-bastard)? It converts a vss repo to git (not very well, but well enough), ignoring the VSS "passwords."

[SoftTalker]

IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.

The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.

[masklinn]

> IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.

IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).

I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).

[heywire]

We still use VSS for a couple codebases. We’re a small enough team that conflicts are rare, but the occasional Teams message “hey, let me know when you’re done with module.cpp” is not unheard of.

[varjag]

Fascinating. An anachronism worthy of steampunk novels!

[cebert]

I’m impressed VSS still works. I used it as part of my first professional software engineering job in 2008 and it felt old then.

[jsrcout]

I'm so sorry. We were using it in the late 90s, and on a monthly basis it would be down for a day or two while the admin plunged out the database. Good times.

[heywire]

It surprisingly works well. We do use a front end to it called VssConnect, but to be honest, I really don’t have any complaints.

[alt187]

That's just maddening enough to start a cult!

The Rube-Goldberg VCS-mutex is hilarious though.

[sir-dingleberry]

This is insane. Thanks for the post.

[whatever1]

I bet now they deal with broken pipelines and dependency hell.

Tools do not fix bad design.