[megous]

At $29 * 7 * 12 per year it became way cheaper for us to just piece together functionality we needed from nginx+cgit+homegrown database to store users/repos/acl/push info and a few git hooks written in a few lines of PHP. The cost is now independent of the number of developers using the system.

So far it did cost ~1 month of paying for github in dev time and I can't imagine it costing much more when we'll want to add some automation on top of the list of accepted pushes/ref updates, for which we did not have a need for so far.

Certainly beats installing 1GiB debian package of selfhosted gitlab and having to figure out why some stupid ruby service is eating increasing amounts of hundreds of MiB of RAM on an empty gitlab instance while doing nothing at all.

That's $2.5k/yr that can be put into something else.

When developers do operations... I guess. :D

[mardifoufs]

I mean I dislike the gitlab pricing increases, and we are kind of regretting sticking with them (especially since everyone on the Gitlab instance has to pay for a seat, even when not using any of the premium features) considering how Github is cheaper and has a higher release velocity at this point... But to be very fair, we never saw what you describe about memory leaks or slow ruby services. Migrations are also usually silent and done in the background. I guess the software itself is pretty heavy, but predictably so (RAM usage rarely spikes, for example).

We have a 400+ seats instance, with a decade worth of code and it just works 99.99% of the time, and upgrades are generally painless. Though our instance is pretty vanilla, and internal to our corporation. The runners are more finnicky and buggier, especially since we have tons of different build targets but that's besides the point if we are just talking about using it as a dumb git repo. And if you only need it for that purpose why even pay instead of using the community edition?

(I'm in the AI team but I sometimes help the devops team that takes care of the instance to customize our AI pipelines, so I'm familiar with gitlab and with my colleagues experience with it)

[cwsx]

The company I'm currently at uses a self hosted instance, which due to our "security"* constraints we're grossly behind in updates. Not to say that's Gitlabs fault at all, but the amount of issues we have near daily is hilarious (if it didn't simultanously crush productivity).

That said, I recall issues with the Gitlab Enterprise Support / whoever we contact about our license etc, mostly to do with slow or poor communication. Requests would sit for a month before we got a reply, often dismissive or generally unhelpful, though we _were_ on an outdated version so I don't blame them.

I vaguely recall an email about support ending on self hosted instances? I can't recall the details, but I know it triggered an internal investigation into moving away from Gitlab. EDIT: Pretty sure I'm remembering self hosted Jira, a quick Google search shows Jira EOL but no Gitlab.

All of that said, I largely blame my company for the failures here. I wouldn't expect any company to support self hosted, outdated versions. The support issues were annoying, but I'm also not sure how much I can blame Gitlab for that. I'm also struggling to remember the details, so take this as one mans hazy annecdote.

* film industry, security by obscurity in the worst ways, leads to incredibly outdated and neglected tech (we only _just_ transitioned to Python3, and that's only the core services)

[jmholla]

At least parts of what you say ring true in my experience following GitLab releases more closely (e.g. 1-2 months behind). I would highly recommend not self-hosting if you're going to go with GitLab. Performance issues will appear and disappear between updates, and sometimes on a whim and you're never quite sure what you did or didn't do.

I did find the support staff to be fairly responsive, but most of that time felt like me collecting diagnostic information with little actionable material, and sometimes I would have to explain the same thing multiple times in the same support ticket because it switched hands.

If you do still opt to self-host, dig into their documentation: there are little nuggets and hacks they use internally that you'll want to use to get the right performance out of it.

[cwsx]

I absolutely would not opt to self-host, but unfortunately that's not my department. I agree about the performance issues, most of our issues were performance related and did seem quite random (though unsure if that was our self hosted instance or a Gitlab issue, sounds like it could be both).

I will admit I'm probably overly harsh on the support staff, and misrepresenting the support issues we had. I wasn't directly involved in most of it so I'm parroting what I've heard from coworkers that were more involved, which isn't the whole story. Though the times I was directly involved (in support requests) the experience mostly matched yours, with a couple (albeit rare) cases of slow replies.

In terms of self hosting I 100% agree, and anyone who is thinking of setting up their own self hosted instance should take note of your comment.

[mardifoufs]

Self hosting is fine... but only for an internal instance. The release cycle is pretty extreme, with a critical security update seemingly popping up every two weeks. I mean that's better than not patching the issues, but it still means you have to stay on top of it. Having an internal/private network instance doesn't actually help you all that much, but it still gives you a little more breathing room.

(and I know it might contradict my earlier comment saying that Github's release velocity is a plus, but it doesn't. Most gitlab releases don't introduce useful features, they mostly patch security issues and regressions. For example, the Runners are in a dire need of tons of features and an outright rework of some parts, but barely get any. Which is sad since they were so far ahead of the competition not so long ago.)

[jmholla]

> The release cycle is pretty extreme, with a critical security update seemingly popping up every two weeks.

That just reminded me of my least favorite thing about their releases: they brag about releasig on time for however many months in a row, but they're always quickly followed up with bug and security fixes. I felt their presentation of consistency was misleading at best.

[megous]

My bad experience was mostly just with installing gitlab on a 1GiB RAM VM, to see how it will fare and how easy it is to manage. I expect it to work for people who don't mind throwing a 16GiB+ RAM machine at it. But our dataset is currently like ~200 MiB and simple relatively dumb git hosting just works much better in our case.

[jcrawfordor]

GitLab should run fine on a machine with 4GB memory - this is the smallest recommended memory allocation, spec'd for up to 500 users. 2GB tends to work okay for testing but 1GB is indeed probably too small for all of the services to start. Postgres actually tends to be the long pole in the tent on small systems.

[deng]

Sorry, but the recommended system requirements are an absolute joke. They might be OK if you really only do pure code hosting and don't use CI, container/package registries, project planning features and whatnot, but on our instance with 400 users, the sidekiq background jobs alone easily eat 12GB RAM (I had to extend sidekiq to 4 processes just to deal with the load, otherwise GitLab would become unresponsive).

[onion2k]

The problem with a homegrown system is that every developer working within it will see it as a barrier rather than a cost saving. Whenever they can't do something that's trivial on Github/Gitlab it'll be a reminder that the company would rather save $xxx/year than make their job easier.

Ultimately, having a simple, well-integrated, industry standard stack that includes the same tooling every other company uses is a perk of working at a well-funded or profitable company. People leave for less.

[hamandcheese]

> it'll be a reminder that the company would rather save $xxx/year than make their job easier.

My company has already made this crystal clear by switching to MS Teams.

[cwsx]

Every morning we begin stand-up with a ritual, our whole team simultaneously muttering something arcane whilst manipulating the inner workings of Teams, hoping to appease the MS gods.

Shortly after the ritual is complete you hear a cacophony of "fucking teams wouldn't find my headset/camera", and the day begins.

[midoridensha]

Your team should have a strict policy to not do ANY work at all each day until the stand-up is complete, and not to do the stand-up until Teams is working correctly for every team member, no matter how long this takes.

[cwsx]

If only I could, I could use some time off...

[kiratp]

A developer costs about 250K a year all up. That doesn’t include opportunity cost.

49, 40 hour weeks a year = $127.55 an hour.

29* 7 * 12 / (127.55) = 19.09 hours

If it takes you more than 2.3 dev days (100% productive) you’re negative ROI doing it yourself.

This math doesn’t even factor in the opportunity cost of doing this.

[skeeter2020]

>> A developer costs about 250K a year all up.

Gotta love HN math. Calculations to two decimal places, starting with a number plucked from the high end of a distribution with standard deviation of at least 100K.

[jdminhbg]

> the high end of a distribution

What a dev costs is not the same as what an advertised salary for a dev is. This is closer to the middle of the distribution for dev costs in the US.

[kiratp]

Ok, so at 80K a year, that’s a 3x multiplier.

So annually you have to spend < 3*19 = 57 hours on your custom built source control and CI to come out on top.

Can’t be done outside of “we hired the OG devs of Gitlab/hub/etc”

[grogenaut]

don't forget benefits, hardware, support staff portion, and possibly office. We typically calculate 50% over... so that's 120k at a likely low end.

we run ghe at work and I know we spend 8 hours 4+ times a year for a test upgrade and then an upgrade from an eng that makes above either of your estimates in salary alone.

None of this includes all the work we're not doing to build new integrations or features in the ci system that we get for pay for the product. But we're not a scrappy startup either. We're paying down much of the tech debt from being a scrappy startup, it's not been cheap.

[mettamage]

> A developer costs about 250K a year all up.

I currently cost more like 80K in euro's if I reason from my employer's side. So tell me, how are you getting to the 250K exactly?

Europe exists as well and even in the US there are enough companies that don't pay FAANG salaries.

[trowawee]

Even if you throw out SF salaries as a wild outlier, this isn't actually that ridiculous. An average quality mid-career dev (5-10 years exp.) in a second tier market like Chicago/Denver/Austin/Boston can pretty easily make 170-200 in cash. "[A]ll up" is the key here tho; there's a big non-salary component in the US that doesn't exist in Europe. Tack on health insurance and the total cost to the company will easily blow past 250 right there. Plus, you're probably giving them some equity and a yearly bonus.

I'll be honest, it does sometimes blow my mind to see how low salaries are over there when I look at job postings and Who's Hiring and the like. I'm jealous of a lot of things Europeans take for granted, but it's wild to me to see senior positions in major European capitals paying the amount that I made two years out of a bootcamp.

[MrFoof]

Thing is, even in those 2nd tier markets, you're still looking more towards the top. I say this as someone having spreadsheets "I shouldn't have" from folks towards the top of acquisitions into larger companies that do pay well. Companies you've at least heard of, but certainly aren't disrupting a market. No, they're not the ones at the top of your list, but a fair number you'd consider respectable.

There's a lot of companies paying 2/3rds of that cash.

Then there's the bad ones, that are still trying to pay half. Seriously. They're not places you even realize exist, and if you went in for an interview you'd instantly sense you're not where you want to be. Places where you watch managers basically bully interviewees to look for immediate subservience, because they want to ensure they'll "yessir" without hesitation. These places absolutely exist. In all of those markets. I've seen quite a number of them.

-- -----

I know, this isn't what we all try to aspire to here, but I say this as someone with far too much experience with Boston (very specifically), Denver and Austin over the past 20+ years. I've had the "good but not amazing" and many "bad" companies as clients of mine. I've talked to their staff. I've done my damnedest to ensure the good eggs know where they stand in the market, and help them move on if they wanted to.

HN very much looks at 75th percentile on up. But once you go down the ladder in the compensation offered, you'll encounter a ton of people where $170-200K salary or salary+bonus would be a 20-30% bump in their comp in those markets. Then you get to the bad places, where they're legit making half and feel thankful for it.

[jdminhbg]

US salaries don’t include employer payroll taxes, unemployment insurance, health insurance, retirement matching, etc. Total cost can easily be 2x advertised salary.

[meghan_rain]

Don't tell them it becomes 40k after taxes or people here won't believe you.

[Petersipoi]

Software Engineers work for $40k net in Europe? Genuinely asking.

[deng]

Yes. In Germany, if you are a good earner, you pay 42% taxes, plus obligatory healthcare about 900EUR/month, pension fund and unemployment insurance (also all obligatory). So if you earn 100k (which is already a high salary in Germany), you usually end up with less than half net. I know this sounds crazy to US citizens, but this system comes with several advantages which I wouldn't want to miss.

[mettamage]

To be honest, I feel Europeans are simply being underpaid compared to US FAANG salaries. Not necessarily compared to EU FAANG salaries as they are lower and even if that's case, it's not glaringly obvious (to me at least).

[mettamage]

Yep, me, there's nothing that pays more to be honest. If you're hiring and it pays significantly more, I'm interested.

[siva7]

Yes. And that's already mid-career salary.

[XajniN]

Only if they have a good salary

[strken]

A developer:

in the US

in specific rich cities in California, New York, Texas, Washington, and Colorado

at a handful of tech companies which are currently bedevilled by lay-offs

averaged across all roles

GitLab doesn't have regional pricing.

[tjoff]

That assume that gitlab is friction free and takes zero time to use?

Maybe something like gitea+teamcity solution is better and cheaper and has less opportunity cost?

I like gitlab, but it feels they have been walking in the wrong direction for many many years and the consequences are starting to pile up.

[gbourne]

I think the point is too many times developers are a penny wise and a pound foolish when it comes to build vs buy. Because they can build, and it is often fun if you haven't done it before.

Though I often don't take my own advice, I try to ask is what I'm about to build core to our ROI/product, is there an existing solution that gets us 80% there, and yes, what is the cost.

If it is not core to the business, won't drive revenue, and cost are not outrageous, which is a company by company truth, then I much prefer to spend X time building new product than recreating an existing product/tool.

One final point - don't forget to consider the maintenance costs, which in the long term often greater then the initial investment. If your CI goes down, you just blew away and savings you were planning on.

[lazypenguin]

There are also maintenance costs with the off the shelf solutions. E.g. Gitlab self-hosted runners don’t handle very large artifacts well and frequently caused CI to timeout. We had to roll our own artifact management system.

In general I agree with you off-the-shelf is usually better but sometimes a custom tailored solution rather than a generic system can be better.

[zeusk]

Their solution is not a monthly recurring event.

[kiratp]

Their solution will require a developer to ssh into VMs to update, debug, upscale etc.

[megous]

That's about ~1-2 hours a year and already has to be done, because that VM has been used to run other dev services. Also with simple solutions, the benefit is that there's usually nothing to debug. It just works.

And the cost of that has to be compared to cost of gitlab subscription and other gitlab related headaches.

[jayd16]

Yeah, now its 24/7 ops headache.

[zeusk]

like gitlab isn't?

[megous]

Hehe, even at US prices we'd break even after half a year.

Average salary where I live is $21k/yr with average devs taking maybe $25-40k/yr. Top jobs being ~$65k

[meghan_rain]

The classic HN Dropbox comment, never gets old.

[deathanatos]

… sometimes, I do think it's merited. We switched to Github Actions for our CI and it's been … mixed. It was a hell of a lot of work to switch (GHA is significantly different than what we were moving from and … buggy) and even then, some of the pricing on GHA high. 24¢/GiB/mo for storage[1] which is like 8×? 10×? the sort of cloud baseline, 50¢/GiB data transfer [2], and the runners are similarly costly, IMO. I'm not sure if GHA wins out, in part because we're not done switching.

You can do self-hosted runners aaaand… welcome to infra/ops?

A little NIH syndrome would do the industry good. We need more stuff invented. The current stuff is mediocre.

[1]: https://docs.github.com/en/billing/managing-billing-for-gith...

[2]: I think it's only egress, and only from packages though?

[megous]

Yes, but all we used gitlab for was git repo hosting with push access control for about 40 repositories to allow sharing code via push/pull and not much else (almost no issues/pull requests). It's ridiculous to pay $2500/yr so that 7 people can do push/pull over slow growing set of ~200MiB of data when it can be handled in much more performant way by a 36$/yr (+ a small one time setup cost) VM we already had for running other things.

[koolba]

Why were you paying for it then?

GitLab has a CE version that’s free for commercial use. It sounds like you just want a pretty interface for git repos with some merge flow.

[megous]

Running gitlab yourself is just too expensive. Too many docs to read, hard to understand, just too much software in general (just compressed deb package alone is 1GiB in size), hard to backup, hard to debug issues, would require a new separate much more expensive VM, etc. It does weird shit during installation, like contacting letsencrypt and trying to fetch a certificate and failing if the machine is IPv6 only, etc. etc. Recovering from that is not documented anywhere. Too many moving parts. Just sort of blergh for people who like simple solutions.

It doesn't sound like a 1-2hr/year self support software.

[argella]

Why don’t you just run gitlab community edition or gitea

[megous]

Gitlab is massive piece of software, and has inordinate HW requirements for the job we'd be using it for. Also it would not run on the VM we already have set up for other services, backed up, etc.

Gitea is fun and all, but it's not in debian repo, so would require manual updates. Annoying.

[conanbatt]

2.5k doesn't pay 1 day of a developer in a top-of-market startup.

[ElCapitanMarkla]

I need to up my rate :D