[gregwebs]

Gitlab pricing used to be something like $4/mo and they saw incredible growth. They raised their rates to $19/mo and now are seeing slow growth. In April they are raising to $29/mo- not sure what they think is going to happen. Also, for anyone that actively watches, they have to wonder if there will be another price hike once you are on their platform.

Github Enterprise is only $21/mo and for most users it has all the same features of Gitlab.

Gitlab's main differentiator for a long time was CI, but now Github has its own equivalent (Actions).

It seems that Gitlab is only going to be left with 1) existing Gitlab users 2) users that want some enterprise feature set that doesn't exist on Github, and want it in a single platform without a third party

[megous]

At $29 * 7 * 12 per year it became way cheaper for us to just piece together functionality we needed from nginx+cgit+homegrown database to store users/repos/acl/push info and a few git hooks written in a few lines of PHP. The cost is now independent of the number of developers using the system.

So far it did cost ~1 month of paying for github in dev time and I can't imagine it costing much more when we'll want to add some automation on top of the list of accepted pushes/ref updates, for which we did not have a need for so far.

Certainly beats installing 1GiB debian package of selfhosted gitlab and having to figure out why some stupid ruby service is eating increasing amounts of hundreds of MiB of RAM on an empty gitlab instance while doing nothing at all.

That's $2.5k/yr that can be put into something else.

When developers do operations... I guess. :D

[mardifoufs]

I mean I dislike the gitlab pricing increases, and we are kind of regretting sticking with them (especially since everyone on the Gitlab instance has to pay for a seat, even when not using any of the premium features) considering how Github is cheaper and has a higher release velocity at this point... But to be very fair, we never saw what you describe about memory leaks or slow ruby services. Migrations are also usually silent and done in the background. I guess the software itself is pretty heavy, but predictably so (RAM usage rarely spikes, for example).

We have a 400+ seats instance, with a decade worth of code and it just works 99.99% of the time, and upgrades are generally painless. Though our instance is pretty vanilla, and internal to our corporation. The runners are more finnicky and buggier, especially since we have tons of different build targets but that's besides the point if we are just talking about using it as a dumb git repo. And if you only need it for that purpose why even pay instead of using the community edition?

(I'm in the AI team but I sometimes help the devops team that takes care of the instance to customize our AI pipelines, so I'm familiar with gitlab and with my colleagues experience with it)

[cwsx]

The company I'm currently at uses a self hosted instance, which due to our "security"* constraints we're grossly behind in updates. Not to say that's Gitlabs fault at all, but the amount of issues we have near daily is hilarious (if it didn't simultanously crush productivity).

That said, I recall issues with the Gitlab Enterprise Support / whoever we contact about our license etc, mostly to do with slow or poor communication. Requests would sit for a month before we got a reply, often dismissive or generally unhelpful, though we _were_ on an outdated version so I don't blame them.

I vaguely recall an email about support ending on self hosted instances? I can't recall the details, but I know it triggered an internal investigation into moving away from Gitlab. EDIT: Pretty sure I'm remembering self hosted Jira, a quick Google search shows Jira EOL but no Gitlab.

All of that said, I largely blame my company for the failures here. I wouldn't expect any company to support self hosted, outdated versions. The support issues were annoying, but I'm also not sure how much I can blame Gitlab for that. I'm also struggling to remember the details, so take this as one mans hazy annecdote.

* film industry, security by obscurity in the worst ways, leads to incredibly outdated and neglected tech (we only _just_ transitioned to Python3, and that's only the core services)

[jmholla]

At least parts of what you say ring true in my experience following GitLab releases more closely (e.g. 1-2 months behind). I would highly recommend not self-hosting if you're going to go with GitLab. Performance issues will appear and disappear between updates, and sometimes on a whim and you're never quite sure what you did or didn't do.

I did find the support staff to be fairly responsive, but most of that time felt like me collecting diagnostic information with little actionable material, and sometimes I would have to explain the same thing multiple times in the same support ticket because it switched hands.

If you do still opt to self-host, dig into their documentation: there are little nuggets and hacks they use internally that you'll want to use to get the right performance out of it.

[cwsx]

I absolutely would not opt to self-host, but unfortunately that's not my department. I agree about the performance issues, most of our issues were performance related and did seem quite random (though unsure if that was our self hosted instance or a Gitlab issue, sounds like it could be both).

I will admit I'm probably overly harsh on the support staff, and misrepresenting the support issues we had. I wasn't directly involved in most of it so I'm parroting what I've heard from coworkers that were more involved, which isn't the whole story. Though the times I was directly involved (in support requests) the experience mostly matched yours, with a couple (albeit rare) cases of slow replies.

In terms of self hosting I 100% agree, and anyone who is thinking of setting up their own self hosted instance should take note of your comment.

[mardifoufs]

Self hosting is fine... but only for an internal instance. The release cycle is pretty extreme, with a critical security update seemingly popping up every two weeks. I mean that's better than not patching the issues, but it still means you have to stay on top of it. Having an internal/private network instance doesn't actually help you all that much, but it still gives you a little more breathing room.

(and I know it might contradict my earlier comment saying that Github's release velocity is a plus, but it doesn't. Most gitlab releases don't introduce useful features, they mostly patch security issues and regressions. For example, the Runners are in a dire need of tons of features and an outright rework of some parts, but barely get any. Which is sad since they were so far ahead of the competition not so long ago.)

[jmholla]

> The release cycle is pretty extreme, with a critical security update seemingly popping up every two weeks.

That just reminded me of my least favorite thing about their releases: they brag about releasig on time for however many months in a row, but they're always quickly followed up with bug and security fixes. I felt their presentation of consistency was misleading at best.

[megous]

My bad experience was mostly just with installing gitlab on a 1GiB RAM VM, to see how it will fare and how easy it is to manage. I expect it to work for people who don't mind throwing a 16GiB+ RAM machine at it. But our dataset is currently like ~200 MiB and simple relatively dumb git hosting just works much better in our case.

[jcrawfordor]

GitLab should run fine on a machine with 4GB memory - this is the smallest recommended memory allocation, spec'd for up to 500 users. 2GB tends to work okay for testing but 1GB is indeed probably too small for all of the services to start. Postgres actually tends to be the long pole in the tent on small systems.

[deng]

Sorry, but the recommended system requirements are an absolute joke. They might be OK if you really only do pure code hosting and don't use CI, container/package registries, project planning features and whatnot, but on our instance with 400 users, the sidekiq background jobs alone easily eat 12GB RAM (I had to extend sidekiq to 4 processes just to deal with the load, otherwise GitLab would become unresponsive).

[onion2k]

The problem with a homegrown system is that every developer working within it will see it as a barrier rather than a cost saving. Whenever they can't do something that's trivial on Github/Gitlab it'll be a reminder that the company would rather save $xxx/year than make their job easier.

Ultimately, having a simple, well-integrated, industry standard stack that includes the same tooling every other company uses is a perk of working at a well-funded or profitable company. People leave for less.

[hamandcheese]

> it'll be a reminder that the company would rather save $xxx/year than make their job easier.

My company has already made this crystal clear by switching to MS Teams.

[cwsx]

Every morning we begin stand-up with a ritual, our whole team simultaneously muttering something arcane whilst manipulating the inner workings of Teams, hoping to appease the MS gods.

Shortly after the ritual is complete you hear a cacophony of "fucking teams wouldn't find my headset/camera", and the day begins.

[midoridensha]

Your team should have a strict policy to not do ANY work at all each day until the stand-up is complete, and not to do the stand-up until Teams is working correctly for every team member, no matter how long this takes.

[cwsx]

If only I could, I could use some time off...

[kiratp]

A developer costs about 250K a year all up. That doesn’t include opportunity cost.

49, 40 hour weeks a year = $127.55 an hour.

29* 7 * 12 / (127.55) = 19.09 hours

If it takes you more than 2.3 dev days (100% productive) you’re negative ROI doing it yourself.

This math doesn’t even factor in the opportunity cost of doing this.

[skeeter2020]

>> A developer costs about 250K a year all up.

Gotta love HN math. Calculations to two decimal places, starting with a number plucked from the high end of a distribution with standard deviation of at least 100K.

[jdminhbg]

> the high end of a distribution

What a dev costs is not the same as what an advertised salary for a dev is. This is closer to the middle of the distribution for dev costs in the US.

[kiratp]

Ok, so at 80K a year, that’s a 3x multiplier.

So annually you have to spend < 3*19 = 57 hours on your custom built source control and CI to come out on top.

Can’t be done outside of “we hired the OG devs of Gitlab/hub/etc”

[grogenaut]

don't forget benefits, hardware, support staff portion, and possibly office. We typically calculate 50% over... so that's 120k at a likely low end.

we run ghe at work and I know we spend 8 hours 4+ times a year for a test upgrade and then an upgrade from an eng that makes above either of your estimates in salary alone.

None of this includes all the work we're not doing to build new integrations or features in the ci system that we get for pay for the product. But we're not a scrappy startup either. We're paying down much of the tech debt from being a scrappy startup, it's not been cheap.

[mettamage]

> A developer costs about 250K a year all up.

I currently cost more like 80K in euro's if I reason from my employer's side. So tell me, how are you getting to the 250K exactly?

Europe exists as well and even in the US there are enough companies that don't pay FAANG salaries.

[trowawee]

Even if you throw out SF salaries as a wild outlier, this isn't actually that ridiculous. An average quality mid-career dev (5-10 years exp.) in a second tier market like Chicago/Denver/Austin/Boston can pretty easily make 170-200 in cash. "[A]ll up" is the key here tho; there's a big non-salary component in the US that doesn't exist in Europe. Tack on health insurance and the total cost to the company will easily blow past 250 right there. Plus, you're probably giving them some equity and a yearly bonus.

I'll be honest, it does sometimes blow my mind to see how low salaries are over there when I look at job postings and Who's Hiring and the like. I'm jealous of a lot of things Europeans take for granted, but it's wild to me to see senior positions in major European capitals paying the amount that I made two years out of a bootcamp.

[MrFoof]

Thing is, even in those 2nd tier markets, you're still looking more towards the top. I say this as someone having spreadsheets "I shouldn't have" from folks towards the top of acquisitions into larger companies that do pay well. Companies you've at least heard of, but certainly aren't disrupting a market. No, they're not the ones at the top of your list, but a fair number you'd consider respectable.

There's a lot of companies paying 2/3rds of that cash.

Then there's the bad ones, that are still trying to pay half. Seriously. They're not places you even realize exist, and if you went in for an interview you'd instantly sense you're not where you want to be. Places where you watch managers basically bully interviewees to look for immediate subservience, because they want to ensure they'll "yessir" without hesitation. These places absolutely exist. In all of those markets. I've seen quite a number of them.

-- -----

I know, this isn't what we all try to aspire to here, but I say this as someone with far too much experience with Boston (very specifically), Denver and Austin over the past 20+ years. I've had the "good but not amazing" and many "bad" companies as clients of mine. I've talked to their staff. I've done my damnedest to ensure the good eggs know where they stand in the market, and help them move on if they wanted to.

HN very much looks at 75th percentile on up. But once you go down the ladder in the compensation offered, you'll encounter a ton of people where $170-200K salary or salary+bonus would be a 20-30% bump in their comp in those markets. Then you get to the bad places, where they're legit making half and feel thankful for it.

[jdminhbg]

US salaries don’t include employer payroll taxes, unemployment insurance, health insurance, retirement matching, etc. Total cost can easily be 2x advertised salary.

[meghan_rain]

Don't tell them it becomes 40k after taxes or people here won't believe you.

[Petersipoi]

Software Engineers work for $40k net in Europe? Genuinely asking.

[deng]

Yes. In Germany, if you are a good earner, you pay 42% taxes, plus obligatory healthcare about 900EUR/month, pension fund and unemployment insurance (also all obligatory). So if you earn 100k (which is already a high salary in Germany), you usually end up with less than half net. I know this sounds crazy to US citizens, but this system comes with several advantages which I wouldn't want to miss.

[mettamage]

Yep, me, there's nothing that pays more to be honest. If you're hiring and it pays significantly more, I'm interested.

[siva7]

Yes. And that's already mid-career salary.

[XajniN]

Only if they have a good salary

[strken]

A developer:

in the US

in specific rich cities in California, New York, Texas, Washington, and Colorado

at a handful of tech companies which are currently bedevilled by lay-offs

averaged across all roles

GitLab doesn't have regional pricing.

[tjoff]

That assume that gitlab is friction free and takes zero time to use?

Maybe something like gitea+teamcity solution is better and cheaper and has less opportunity cost?

I like gitlab, but it feels they have been walking in the wrong direction for many many years and the consequences are starting to pile up.

[gbourne]

I think the point is too many times developers are a penny wise and a pound foolish when it comes to build vs buy. Because they can build, and it is often fun if you haven't done it before.

Though I often don't take my own advice, I try to ask is what I'm about to build core to our ROI/product, is there an existing solution that gets us 80% there, and yes, what is the cost.

If it is not core to the business, won't drive revenue, and cost are not outrageous, which is a company by company truth, then I much prefer to spend X time building new product than recreating an existing product/tool.

One final point - don't forget to consider the maintenance costs, which in the long term often greater then the initial investment. If your CI goes down, you just blew away and savings you were planning on.

[lazypenguin]

There are also maintenance costs with the off the shelf solutions. E.g. Gitlab self-hosted runners don’t handle very large artifacts well and frequently caused CI to timeout. We had to roll our own artifact management system.

In general I agree with you off-the-shelf is usually better but sometimes a custom tailored solution rather than a generic system can be better.

[zeusk]

Their solution is not a monthly recurring event.

[kiratp]

Their solution will require a developer to ssh into VMs to update, debug, upscale etc.

[megous]

That's about ~1-2 hours a year and already has to be done, because that VM has been used to run other dev services. Also with simple solutions, the benefit is that there's usually nothing to debug. It just works.

And the cost of that has to be compared to cost of gitlab subscription and other gitlab related headaches.

[jayd16]

Yeah, now its 24/7 ops headache.

[zeusk]

like gitlab isn't?

[megous]

Hehe, even at US prices we'd break even after half a year.

Average salary where I live is $21k/yr with average devs taking maybe $25-40k/yr. Top jobs being ~$65k

[meghan_rain]

The classic HN Dropbox comment, never gets old.

[deathanatos]

… sometimes, I do think it's merited. We switched to Github Actions for our CI and it's been … mixed. It was a hell of a lot of work to switch (GHA is significantly different than what we were moving from and … buggy) and even then, some of the pricing on GHA high. 24¢/GiB/mo for storage[1] which is like 8×? 10×? the sort of cloud baseline, 50¢/GiB data transfer [2], and the runners are similarly costly, IMO. I'm not sure if GHA wins out, in part because we're not done switching.

You can do self-hosted runners aaaand… welcome to infra/ops?

A little NIH syndrome would do the industry good. We need more stuff invented. The current stuff is mediocre.

[1]: https://docs.github.com/en/billing/managing-billing-for-gith...

[2]: I think it's only egress, and only from packages though?

[megous]

Yes, but all we used gitlab for was git repo hosting with push access control for about 40 repositories to allow sharing code via push/pull and not much else (almost no issues/pull requests). It's ridiculous to pay $2500/yr so that 7 people can do push/pull over slow growing set of ~200MiB of data when it can be handled in much more performant way by a 36$/yr (+ a small one time setup cost) VM we already had for running other things.

[koolba]

Why were you paying for it then?

GitLab has a CE version that’s free for commercial use. It sounds like you just want a pretty interface for git repos with some merge flow.

[megous]

Running gitlab yourself is just too expensive. Too many docs to read, hard to understand, just too much software in general (just compressed deb package alone is 1GiB in size), hard to backup, hard to debug issues, would require a new separate much more expensive VM, etc. It does weird shit during installation, like contacting letsencrypt and trying to fetch a certificate and failing if the machine is IPv6 only, etc. etc. Recovering from that is not documented anywhere. Too many moving parts. Just sort of blergh for people who like simple solutions.

It doesn't sound like a 1-2hr/year self support software.

[argella]

Why don’t you just run gitlab community edition or gitea

[megous]

Gitlab is massive piece of software, and has inordinate HW requirements for the job we'd be using it for. Also it would not run on the VM we already have set up for other services, backed up, etc.

Gitea is fun and all, but it's not in debian repo, so would require manual updates. Annoying.

[conanbatt]

2.5k doesn't pay 1 day of a developer in a top-of-market startup.

[ElCapitanMarkla]

I need to up my rate :D

[suddenclarity]

I reckon their growth was mostly killed by Microsoft purchasing GitHub and making private repositories free. Most of the open source community seem to be on GitHub anyway so it just removed any incitament for average people to join Gitlab. Once you're invested in one environment it takes something special to move.

[nicolaslem]

This. A while ago Gitlab was really the better option: mostly free and more features (CI, project management...). Github caught up on features and price so why bother going against the network effect when the alternative is not even better?

[chainwax]

Unless Actions has gotten a lot more fleshed out since I last used it, I don't think it's as robust as Gitlab CI.

[stabbles]

- Github Action's design is much better, just look at [1] for example, there's a lot of low-hanging fruit in Gitlab that's not done for some reason

- I've had tons of issues with their kubernetes runners, hard to debug spurious system failures

- They still don't display the last N bytes of the logs when logs get long, just the first N bytes, which is entirely useless when a build fails

- It's awfully slow to navigate through pipelines, in particular child pipelines

- The `stages` things is just useless noise, I just wanna specify dependencies please, I don't want to be forced to also put each job in a stage, it's weird.

- You can't expose a secret only to a single job step; they don't really have finer granularity for job steps like Github Actions has -- they just generate a script.sh that's executed directly.

- Github has many more reusable actions from the community, I don't think Gitlab has that at all? At least not in a convenient way.

[1] https://gitlab.com/gitlab-org/gitlab/-/issues/395964

[hardwaresofton]

> Github has many more reusable actions from the community, I don't think Gitlab has that at all? At least not in a convenient way.

This is the big one.

Github’s ability and dominance in the social space means they have the most active actions ecosystems.

They made actions easier to import share and built marketplaces and now gobs of developers build value into it just for clout (and because they enjoy software of course).

If you think about it carefully, it’s actually a security nightmare. You have no idea what a random Github action you “uses:”d is doing, but in practice 90%+ of the time it doesn’t matter, you get good functionality and save time for free.

[mardifoufs]

YES! We are getting awfully tempted to move at least my AI team to Github. We don't want to dedicate someone to figure out CI/CD/MLops pipelines, which would mostly amount to translating existing, published, official github action pipelines to something that can work on Gitlab. Our devops teams can help, but when in our case we need a pretty domain specific pipeline setup. Most of the pipeline related docs for Azure ML, for example, is based around very extensive github actions templates. It would take minutes to just deploy it there.

I mean in this particular case, it's unfair since microsoft would obviously favor github. But it's not like there's a marketplace where a community made version could be used as a replacement...

[graton]

> The `stages` things is just useless noise, I just wanna specify dependencies please, I don't want to be forced to also put each job in a stage, it's weird.

You don't have to. GitLab supports dependencies. Using the `needs:` keyword.

It was released in August of 2019.

https://about.gitlab.com/releases/2019/08/22/gitlab-12-2-rel...

[stabbles]

You still need both `needs: ...` and `stage: ...`

> In GitLab 14.0 and older, you can only refer to jobs in earlier stages.

> In GitLab 14.1 and later you can refer to jobs in the same stage as the job you are configuring.

Why bother with stages at all?

[mardifoufs]

Oh my god their kubernetes runners are a nightmare sometimes. Even the configuration is arcane and the parameters can overlap, or not. It's crazy.

[jamesfinlayson]

> The `stages` things is just useless noise, I just wanna specify dependencies please, I don't want to be forced to also put each job in a stage, it's weird.

I think there a few things like this in GitLab (like the ability to run pipelines against both commits and pull requests) - GitLab give you a lot of flexibility... but so far it's all just been extra bother to deal with.

[Aeolun]

I used to think this, but after having gotten used to Github Actions I’m moving everything over.

Ultimately I constantly felt like everything was half baked when using Gitlab. I still kind of feel that way with Github, but to a much lesser extend.

More importantly, they allow me to pay my $21/month monthly which means they get my business instead of ‘only pay per year’ Gitlab.

[paulddraper]

GitHub actions are great.

Gitlab CI is a bunch of weird hard to understand bolt on conf, undergirded by a defunct docker-machine project

I tried debugging some problems I was having with a runners. Turns out the default Ubuntu AMI was like 6 years old. Does anyone use this thing?

[prepend]

I disagree. Actions is really nice and I don’t have to set up and debug runners. In GitHub, I can have dedicated runners if I want, but99% of the time, I just use vanilla Ubuntu runners.

My fear though is that GitHub will Jack up their prices once I’m running everything. Right now they give 50k minutes per month, but that could change at any point. GitHub enterprise started out at $8/month and now it’s $20+.

[memco]

> Github Enterprise is only $21/mo and for most users it has all the same features of Gitlab.

Of those things that I like more in Gitlab, it is deploy tokens: an easy way to be able to dole out permission to clone code where needed without having to bother with key management or user management. As far as I can tell GitHub has something like this but it requires setting up an app to talk to the API and isn’t nearly simple as the button available in Gitlab.

[brasic]

Not sure what you mean. Adding a deploy key is about as simple as I can imagine, definitely doesn’t require messing with apps, which I agree is a pain.

https://docs.github.com/en/authentication/connecting-to-gith...

(disclosure, former GH)

[memco]

That's a different thing. Here's some info on deploy tokens: https://docs.gitlab.com/ee/user/project/deploy_tokens/. It's a one-off username+password pair that can be used for HTTP cloning without setting up SSH keys or creating users. As far as I can tell in Github you use either automatic tokens[1] or personal tokens[2]. Personal tokens look easier to set up, but they are tied to a user and not the repo or organization, which isn't great if you're using this for a company—you have to create a whole separate token user if you don't want it tied to someone's personal account. Automatic tokens require a script to manage.

As nice as this one feature of Gitlab is it is not a major show-stopper. Almost everything we need is available in most git services. So to GGP's point: we are likely to go for the most affordable option even if there's some tradeoffs.

[1]: https://docs.github.com/en/actions/security-guides/automatic... [2]: https://docs.github.com/en/authentication/keeping-your-accou...

[mau]

I think the GH app is a workaround for this:

> Deploy keys only grant access to a single repository. More complex projects may have many repositories to pull to the same server.

[brightball]

$29 / month as it relates to developer salaries is not impactful.

[speedgoose]

It adds up. 29$ for GitHub, 19$ for GitHub Copilot, 20$ for ChatGPT Plus, a few codespaces and runners, perhaps 7.5$ for the sadomasochist project managers who wants Jira, and you may get a nice bill at the end of the month.

And then you have people like GitKrakken who ask 19$ for a git client, or Teleport who ask 36$ for a ssh client.

[brightball]

It does, but you have to associate these cost as part of the employee expense and not a separate budget line item. An extra $100 / month for tools needed by somebody being paid $5,000+ / month is nothing.

[msm_]

In my previous organisation $19/month (IIRC) was a dealbreaker. That was non-US so salaries are lower, and suddenly I had to explain why we absolutely needed the paid version so much (instead of sticking to the free one). To make it worse, most of the people that had access to gitlab were researchers/managers/non-technical people and not actually developers, so only a few people would benefit from the paid version. So we never bought the paid version.

[mschuster91]

> It seems that Gitlab is only going to be left with 1) existing Gitlab users 2) users that want some enterprise feature set that doesn't exist on Github, and want it in a single platform without a third party

Add in 3) people who want a reasonable option to self-host that's not charging you to oblivion.

[nunez]

GitLab Auto DevOps and (now) GitLab Ultimate are definitely compelling differentiators that Github doesn't currently have, but I can't see Github just _not_ responding to that in a big way.

[HWR_14]

Or (3) Gitlab users that want to self-host. But while those might be users, they are not great customers.