[parfe]

>GitLab has memory leaks. These memory leaks manifest themselves in long-running processes, such as Unicorn workers. (The Unicorn master process is not known to leak memory, probably because it does not handle user requests.)

>To make these memory leaks manageable, GitLab comes with the unicorn-worker-killer gem. This gem monkey-patches the Unicorn workers to do a memory self-check after every 16 requests. If the memory of the Unicorn worker exceeds a pre-set limit then the worker process exits. *

I know this is a Rails issue, but I cannot believe this is still a problem. I remember when the RoR folks blamed the Rails memory leaks on Zed, and Mongrel, more than a decade ago. They were obviously incorrect. Did the community simply give up on fixing the problem?

* https://docs.gitlab.com/ee/administration/operations/unicorn...

[obl]

I have no idea about either ruby or rails but since ruby is a managed language this kind of issues seem like they should be pretty easy to debug ?

After all, if the gc is not freeing the memory then it must know a reachable path to it so the blame should be easy to assign.

Does ruby not have any good heap profiling tool ?

[skrebbel]

I don't know anything about Rails but most memory leaks in managed languages are stuff like lists growing infinitely large, parts of them pointing to data that is no longer relevant. Think badly implemented caches, stuff like that.

If you have enough not-perfectly-cleanly-separated layers of data management interacting, then this gets hard to debug pretty fast.

[pjmlp]

Not when you have GC graphical debuggers like Java,.NET and JavaScript do.

[briandear]

Why are they still using unicorn over Puma? That’s one question. Using unicorn killer is the sign of some bad engineering. It’s the equivalent of restarting your server every hour.

[sytse]

We are planning to start using multi threading and Puma so we can reduce the memory requirements of GitLab. The last time we shipped that we had to revert due to hard to diagnose errors. We're taking a more incremental approach now.

[qop]

Were you under the impression that Rails has a functioning dev community?

I am very familiar with the rubyverse, and boy do I got news for you.

Rails is to Ruby as the Big Dig is to Boston.

But it's hard to get Rails devs to admit it. They know it, of course. They live it. They breathe bad code, they exist as extensions of a philosophy that has failed it's way to the top.

I used to think Zed was a dramatic weenie. Then I got a bit deeper in Rails to learn how to pursue better exploits. And deeper. And deeper. And finally I just hit a point where I just wasn't willing to work on RoR-related gigs.

Convention over configuration is one of those ideas that goes great on paper. And then you start in on some buzzwords and fancy lingo, it's the bread and butter of selling to project managers. But there is a point where CoC breaks down at a technical level, and that point is literally the first moment you come across something difficult to express

Rails is weak in doing things ANY other way than the Rails way, and what developers do (and it's not their fault) is using their creativity and ingenuity to solve the problem. I believe this is why metaprogramming in Ruby has such a vocal backing, because there are millions of shims holding up millions of Rails' shitty choices that depend on method_missing fuckery or string-based define_method gadgets. In the spirit of what it means to solve hacker-ish problems, it's great fun to program like that for a while, and then a year or two down the line you look at that shit and it can really break you down. Good developers mistake themselves for bad developers because of this, to be honest.

Zed's probably had the idea to write an entire book of how insanely FUCKED Rails actually is, but to make the book plausibly thorough would mean a human being being exposed to far more stupidity than modern science can safely treat for. It'd be dangerous to have that much stupid intake just for a book.

That unicorn-worker-killer gem is a perfect caricature of the kinda problem-solving you run into with Rails. Let it die a horrible death, restart that shit with some logic to let it keep dying predictably.

That technique, multiplied by every component in the system, multiplied by every developer on the team, multiplied by MVC requiring developers to have to know too broadly the system they're working on, multiplied by however many years and Rails versions the project has somehow survived, you approach something I can only imagine is quite similar to what hell must be.

[artursapek]

I spent 3 years working on a Rails app and it was enough to make me swear off Rails for the rest of my life. I think the Rails dev community has good intentions, and the design of Rails is actually pretty nice. The problem in my opinion is more with Ruby, and human laziness. Ruby lets programmers do absolutely anything they can think of no matter how unexpected or un-intuitive it will be to their colleagues. Combined with the rigid conventions of Rails' design, this inevitably leads to a maintenance disaster with people, like you say, getting too creative.

Refactoring a large Ruby on Rails codebase is basically impossible. There's no compiler to help you along; hell the code is evaluated lazily. So instead of refactoring, people monkey-patch their problems away. It's horrible, really really horrible.

I'm now getting flashbacks to trying to understand how files get loaded in that codebase... I need a drink

[joelbluminator]

I don't completely get this sentiment. Is refactoring a big java program easy? Yes, you will get nice hints from the IDE, but who knows how many things you screwed in the process of refactoring that the compiler isn't telling you about? Especially when it's not your code, when the person who wrote it isn't there, when you're not 100% what the code is actually doing, when it's undocumented and the intent isn't clear etc etc. Or worse when there aren't any tests which I'm sure happens a lot in java land. Why would this be specific to Rails? You're making it sound as if the twitter team can rewrite their app in 2 days because of 'compiler'. Big complex projects are big and complex. In Java / C# / Scala they're even bigger because of types, interfaces, traits and other kind of verbosity. It makes it harder to wrap your mind around the application in the first place.

[nerdwaller]

It seems like you’re taking this a bit further than the parent comment said - e.g. there’s no claim that any major piece of software can be refactored that quickly, however the general process can be made simpler and safer with decent static tooling.

As a concrete example on a small scale of something common such as renaming a model field name. Given that ActiveRecord reflects the database and generates the methods dynamically there’s pretty much no good way to refactor that field easily. It requires a lot of manual searching/filtering to find possible usages and then time on that to ensure it’s not a shared name on a different object type. From what I’ve seen in many code bases it ends up manifesting as `delegate` usage and often just expands the API exposed by the model.

Compare that to something compiled, or even something like python that doesn’t encourage meta too much, and you can pretty much find all symbol usages with whatever intellisense engine you prefer (ctags, JetBrains, jedi, ...)

[joelbluminator]

I agree static analysis can help. Specifically for changing column names it just doesnt really happen that often in most projects I worked on...so I never really thought it to be a big deal.

[StaticRedux]

Then you're lucky. I specialize in refactoring and reorganizing extremely messy and large / complex applications. And Rails is my fucking nightmare. Especially when untangling ActiveRecord callbacks, fat controllers, convention over configuration, magic method, and database changes.

I would give a kidney to have static compilation in rails when refactoring.

[joelbluminator]

I think linters and IDEs will help with that.i already see big improvements in js linters

[nerdwaller]

That goes to my point, rails doesn't have the symbols in code for the editor to use to refactor because they are added at runtime dynamically when ActiveRecord reflects the database. You can have a full model that just has `class User < ActiveRecord::Base; end` that has 15 fields on it, none of which are in the code.

[terracatta]

> They know it, of course. They live it. They breathe bad code, they exist as extensions of a philosophy that has failed it's way to the top

I think this is a dangerous line of thinking. This crosses the line of being productively critical of a language or a framework into actually attacking people.

By promoting this line of thinking, you increase the tolerance of this kind of attack.

People arrive at their personal choice of language and framework as a function of their environment, their education, their first job, what's popular at the time, and myriad of other factors.

I am sure you have preferences for languages and frameworks, and I'm sure there are other people who don't agree with them. Do you want those people giving you reasoned arguments about why you should change your preferences or, do what you did, make allusions that you are so irredeemably stupid that you physically exhale shoddy code?

[mikekchar]

My day job is working on Rails apps (most of the time anyway -- we have a diverse infrastructure). I'm not a big fan of Rails architecture and struggle to communicate to others why I think the "Rails Way" is not necessarily a good way. But these things are hard to swallow, especially if you haven't seen anything else before.

However, I have a special dislike for "convention over configuration". What that really means is: You must write code my style. As there is no configuration, it will be difficult to discover what that means by reading the code. You must study and memorise the one true way. If you deviate from it, strange things will probably happen and it will be your fault for not doing it my way.

To be fair, Rails documentation is really very good. Also, this is a selling point for the framework. If you give no choices then everybody must do it the same way. Even if that way is stupid. You can hire a whole bunch of people who "know rails" and they can struggle along and make progress. You don't have to trust them with decisions because the important decisions were already made for them.

I've had a wide and varied career before doing this work. It's hard for me accept "because it's the Rails Way" to questions like "Why are we doing this when it is causing problems?" I don't mind people making mistakes as long as it is not a thoughtless mistake. "I thought this would work, but I was wrong", is a wonderful sentence in my book. Unfortunately that's the kind of action that many programmers are dissuaded from taking -- instead they are encouraged to do it the "Rails Way".

Lots of beginning programmers also prefer it this way. When you are first starting out (and... well... even when you've been doing it for a couple of decades) it's hard to know which way is better. You might even have a preference, but then struggle greatly when someone asks "Why is that better". Especially beginning programmers often get stuck without knowing any way to accomplish something. In those cases, the "Rails Way" is extremely comforting. 1) It's a way. 2) It will probably work (at least grossly) 3) Famous people say it's good, so it's very, very comforting to agree. People entrenched in the community reinforce each other until it seems practically impossible that there might be a better way. But they may not have looked.

Of course Rails is not the only "Way" out there that espouses these ideas. It's just a currently popular one. What's really at fault is the "convention over configuration" attitude -- Don't think; just do it my way. I hate that.

[keybits]

Still lots of love for Rails in the HN community on this recent thread: https://news.ycombinator.com/item?id=17355187

[CSMastermind]

Rails is popular as startups because it's easy to pick up as a first language meaning the labor market was cheap and flooded with coding boot camp grads. Startups love cheap labor and they don't really care if things are built to last or will scale. They only care about it lasting until their next funding round or (hopefully) their exit.

[joelbluminator]

Rails Devs are well compensated...where are you getting this from? Yes rails is more suitable for rapid development than spring, and easier to teach and learn. How is that bad though

[TheAceOfHearts]

I love Rails for certain projects, but it's definitely true that at a certain point things start to break down. Maintaining a large complex application with Rails is not fun at all. ActiveRecord is amazing, until it's not.

IMO, you can still learn tons of great practices with Rails, and for certain projects I still think it's a great choice... But you have to know when to drop it and move on to more robust solutions.

Like most engineering tools, it makes certain tradeoffs, and you have to know how to pick the right one for your job.

[rimliu]

At certain point everything starts to break down no matter the stack.

[StaticRedux]

You're absolutely right, you're just completely ignoring context. Some things break down a lot sooner. Eating or drinking too much of anything can kill you, even water. But drinking acid will get you there a whole lot faster. Rails is the acid of the programming world.

[ljm]

After many years of working on a variety of Rails projects, I've pretty much come to the same conclusion and, if you look at it another way, it's the same conclusion no matter what you work with.

Rails is incredibly effective for solving the problems that existed at the time it was conceived. CRUD websites, rapid prototypes, things that aren't incredibly complex. Not an exhaustive list, but that's what I reach out to Rails for now.

Once you get past that certain point, not only does the framework fail to offer a convention for the problem you might face, but it also requires a mature team of developers to guide things forward from that point on, and really think about how to build something that Rails itself has no decent answer for. That solution might be to break out of Rails, or switch to raw SQL queries, or to proxy to services that can handle the task better. It's almost never a good idea to double down on the convention at that point.

[9935c101ab17a66]

I am no expert at all (a current comp sci student), but when I read this line in the unicorn-killer explanation I lol'd:

> This is a robust way to handle memory leaks

Seriously wow.

[lloeki]

Not to detract from the Rails situation because I largely agree with it, but there are also issues with Ruby's memory management, as some memory just can't be reclaimed. Rebuilding Ruby with jemalloc helped in mitigating rampant memory fragmentation for our loads and use cases, but things can only go so far regarding how things are managed. At some point the processes need recycling, so yes, most literally, regularly reaping the workers is most literally a robust way to perform an OS-level sort of GC, as in you're 100% sure it works with any reliability, and even though it may hurt some theoretical sensibilities this is a very hands-on, practical solution.

[briandear]

You are blaming Ruby? Plenty of major orgs running Ruby at scale aren’t resorting to restarting the server to solve memory leaks.

It is a practical hands-on solution. However it’s also bad engineering. I too have used unicorn-killer and it was always the result of trying to make bad code not crash our servers. It is a lazy hack solution that I used because it was easy and at the time didn’t know enough to fix the actual issues. I was inexperienced. No disrespect intended but with the suggestion that essentially restarting the server is a robust strategy against memory leaks, it’s kind of hard to be confident of the technical robustness of the platform, especially at scale.

I can guarantee that Github isn’t using unicorn-killer nor blaming Ruby for performance issues.

[briandear]

Agreed. As someone who has worked with Rails over the past 9 years, the only time unicorn-killer is generally used is to save an application from very bad engineering. That’s it. That’s the only reason to ever use it. No serious (or even not-so-serious) production app should be using unicorn-killer as a strategy. Ridiculous fail. Rather than releasing shiny new stuff, it might be wise to fix the core issues with the stability of the system first. Using New Relic or Skylight should at least help narrow down where the memory leak originates.

I know we are supposed to be “Hurray Gitlab! Open core!” but it feels like GitLab support around here is contrived and astroturffed, by GitLab insiders. Any company that actually claims unicorn killer is a robust way to handle memory leaks has some extremely incompetent technical leadership. The robust way is to not to have memory leaks! Essentially restarting your servers all the time (which is what unicorn killer practically does) is a ridiculous way to handle memory leaks.