[sdrwefgfvb]

> they went over how they handle a billion requests a day with mostly a Rails monolith

1 day has 246060=86400 seconds. 1bn/86400 is more than 10000 reqs/second, so each request has to be served in less than 100 us. According to [1], random access on an SSD is about 150us. This suggests to me it's likely that most of these are being served cached from a CDN. Are we supposed to be surprised that this can be done by a rails monolith? We don't know how many of those requests are actually hitting the rails app.

[1] https://gist.github.com/jboner/2841832

[ubertaco]

This math assumes that the requests are serial (each begins and completes before the next one starts), rather than concurrent.

[sdrwefgfvb]

Can you do concurrent reads to a disk? I actually don't know the answer to this. I'm pretty sure you can't on an HDD, but I don't know about an SSD.

[kvirani]

Does each web request to an app server have to hit the disk on the same server? And if so, for the entire request duration? Also, what about horizontal scaling which also means that the db is on a different server (likely with secondary / replicated DBs)

[ekianjo]

The requests are not happening on a single server

[sdrwefgfvb]

I wouldn't call that a monolith then.

[lolinder]

If it's all one codebase that isn't split for deployment, it's still very much a monolith. The distinction between monolith and microservices isn't how many deployments you have, it's how many deployment units. If the app is stateless, replication is strictly an operations concern, so from a developer perspective a single stateless deployment unit is a monolith.

An org might deploy a complete copy of the monolith for a single tenant, and it would still be a monolithic architecture.

[tyre]

Monolith refers to the application. It’s in contrast to a services architecture.

[gregmac]

You can! Well, sort of. (this is new to me, I was curious as well, so thank you)

Native Command Queuing (NCQ) [1] is a SATA extension that lets the drive optimize the order commands are performed in. For HDDs, this means it can do the commands in an order that minimizes overall seeking time. For SSDs, it can concurrently execute commands that are operating on different physical chips.

I can't find a good reference, but it seems like this is also true for RAID volumes, which are almost certainly being used on the server -- the RAID controller can perform parallel reads on each of the independent disks. How well this works is highly dependent on the RAID controller itself.

[1] https://en.wikipedia.org/wiki/Native_Command_Queuing

[KennyBlanken]

If you have to ask such a basic question, you're not qualified to be commenting on how a performance metric isn't impressive.

[rfrey]

Below, you're doubling down on the "monolith == 1 server" point. You should do a bit of research before you continue, the word doesn't mean what you think it means.

[KronisLV]

> You should do a bit of research before you continue, the word doesn't mean what you think it means.

I'd suggest that the word means whatever the majority of people assume it means - that's also how we get the meaning of "agile" to be so inconsistent, depending on which companies/teams/people you talk to. Essentially, people who've only worked with projects that run as single instances might have a pretty different opinion on what a "monolith" is.

I've definitely met a lot of people who'd claim that a single application package across multiple servers is no longer a monolith. The reasoning would probably go along the lines of this: "If a monolith can live on multiple servers, what do you call an application that can only ever live on a single server, with a single instance being launched at the same time?"

So essentially, what would be the names best suited to describe:

  - a single codebase that can only run as a single instance
  - a single codebase that can be deployed with multiple concurrent instances

Personally, I think it might be worthwhile to also answer that, so we have a better idea of what to name things. Otherwise we end up with messes like people taking DDD too far and having a separate microservice per business entity, because they read into the "micro" part of microservices too much.

Oh, also, in regards to the definition of "monolith" that is centered around only how the code itself is deployed, personally I think that a modular monolith ("modulith"? heh) is a great architecture that doesn't get explored nearly enough! A single codebase that is easy to reason about, a single executable that is probably scalable but also simple to deploy however you need, with different parts of the functionality (think front end, different types of API, reporting functionality, file upload functionality etc.) being possible to enable/disable in each of the instances based on feature flags. Want everything on a single instance? Flip all of the flags to "on". Multiple instances? Configure whatever modules you want, where ever you need.

[ajmurmann]

> "If a monolith can live on multiple servers, what do you call an application that can only ever live on a single server, with a single instance being launched at the same time?"

That exists? Are there examples of this, especially once where there is a good reason for this? I cannot even begin to list all the awful issues with this in my head.

[KronisLV]

> That exists? Are there examples of this, especially once where there is a good reason for this? I cannot even begin to list all the awful issues with this in my head.

Most certainly. I'd suggest that many systems out there that ever only needed to run on a single server are structured like this. Even though you could technically take plenty of these systems and launch two parallel instances, you'd get problems because they haven't adopted the "shared nothing" approach, or even just basic statelessness principles.

We tend to forget ourselves with all of our modern and scalable container developments, but there are untold amounts of PHP code out there that stores files and other uploaded data on the very same server, in any number of folders. Of course, you can technically set up a clustered file system, or at least a network based one, unless you are running in a shared hosting environment, in which case you are out of luck.

Oh, and speaking of shared hosting, in theory you should be able to get rid of environments that use cPanel and instead switch to containers, right? Well, no, because workflows are built around it and dozens of sites might be run on the same account with any given shared hosting provider.

You'll be lucky to even find such an environment that has an up to date version of PHP installed and running and resource contention issues will present themselves sooner or later: "Oh hey, this one slow SQL query in this site brings down this other dozen sites. Could you have a look at it?"

I actually helped an acquaintance with that exact problem, I dread myself for agreeing to help because it wasn't a good experience.

Looking at the enterprise space, I've also seen systems out there that store state (e.g. information about business rules) in the actual application memory liberally, as well as things like user session information, because someone didn't know how or couldn't be bothered to set up Redis.

So there an app restart would mean that everyone is logged out. Not only that, but if you have a system which allows users to make some sorts of requests, with business rules about what order they can be accepted in, that means that you can store the output of these states in the DB, but during the processing you have an in-memory queue, which means that you couldn't feasibly have multiple instances running in parallel, because then you'd have a split brain problem. It's like those people had never heard of RabbitMQ while designing it.

Apart from that, there are also issues with scheduled processes. If you've never heard of feature flags or don't see a good reason to use them, you'll run into the situation where you'll have your main application instance executing scheduled tasks in parallel to serving user requests. Worse yet if it's coupled tightly and the application will do "callbacks" for reacting to certain changes, instead of passing the message through the DB or something. Oh and in regards to performance, you better hope that the reporting process you wrote doesn't cause the service's GC to thrash to the point where everything slows down.

Oh, and in addition to that, there are hybrid rendering technologies like PrimeFaces/JSF out there, which store the user's UI state on the server (in memory), whilst sending diffs back and forth, as well as making the client execute JavaScript in the browser for additional interactivity. Think along the lines of GWT, but even more complicated and way worse. A while back some people talked about how the productivity can actually be pretty nice, but what I saw was 100% the opposite, but more importantly there's also no viable way to (easily) distribute this UI state across multiple instances, at least with the way the eldritch monolith is written. I've also seen Vaadin applications with the same problem.

Another factor that can cause situations like this to eventually develop is having a tightly coupled codebase, where you cannot reasonably extract a piece of code into a separate deployment, because it has 20+ dependencies on other services in the app and is called in about 40+ places (not even kidding). While you could try, before you know it you would be sending 20 MB of JSON for simple data fetching calls between applications (again, not kidding - once actually saw close to 100 MB of network traffic between back end services and DB calls for a page to load).

Those are just some of the issues. My suggestion would be to never build systems like that no matter how "simple" they seem and instead just stop being lazy and use Redis, RabbitMQ, or even just PostgreSQL/MySQL/MariaDB tables for ad-hoc queues, anything is better than writing such messes. And if you are ever asked to help someone with anything that starts looking like the above, tell them that your schedule is sadly full or at least very carefully consider your options.

[xorcist]

> because they haven't adopted the "shared nothing" approach,

In practice, many web applications are stateful. The load balancer would see to it that clients keep talking to the same frontend. For larger applications it is important for cache locality.

> untold amounts of PHP code out there that stores files and other uploaded data

This is quite normal when you have some type of blob, and normally what networked file systems are user for.

[KronisLV]

> In practice, many web applications are stateful. The load balancer would see to it that clients keep talking to the same frontend. For larger applications it is important for cache locality.

In regards to front end resources, it shouldn't matter which instance you're talking to, if all web servers are serving copies of the same bundle, given that the resource hashes would match, outside of A/B testing scenarios. It's also nice to explore stateless APIs where possible, and not have to worry about sticky sessions.

In many dynamically scalable setups if you tried talking to API instance #27, you might discover that it is no longer present because the group of instances has been scaled down due to decreased load. Alternatively, you could discover that the instance that you were talking to has crashed and now has been replaced by another one.

Hence, having something like Redis for caching data, or even a cluster of such services becomes pretty important! Of course, there are ways to do this differently, such as taking advantage of CDN capabilities, but for the most part sticky sessions are a dated approach in quite a few cases. It's easier for everyone not to care about ensuring such persistence.

An excellent exception for this: geographically distributed systems where even if you don't care about that exact instance, you still want stuff in this data center to be reached, instead stuff half way across the world.

> This is quite normal when you have some type of blob, and normally what networked file systems are user for.

Nowadays, I'd argue that S3 (or compatibles like MinIO or Zenko) is one of the very few ways to do this properly, or perhaps GridFS in MongoDB - an abstraction on top of the file system, that handles storing and accessing data as necessary. Then, using a distributed or networked file system, or block/object storage (depending on the setup) is a good idea.

However, in general cases, you should never use the file system directly for the storage of your blobs, regardless of whether those are stored locally or in a networked file system, as that is just asking for trouble. Things like maximum files per folder, inode limits, maximum folder nesting/file name length limits, maximum file sizes, writing bad code that allows browsing other directories than the intended ones, the risk of files that might be executed in the case of bad code/configuration, case sensitivity based on the file system, encoding issues, special characters in filenames or directories, need to escape certain characters as well, reserved names in certain file systems and frankly too many issues to list here.

So yes, it is "normal" but that doesn't make it okay, though one also has to understand that often in a shared hosting environment there aren't good options on offer, versus just spinning up a MinIO container and using the S3 library in your app.

[rfrey]

>I'd suggest that the word means whatever the majority of people assume it means

"Monolith" is a term of art in software engineering. You're in a discussion about software engineering. Saying "it means whatever people want it to mean!" is like talking to a bunch of chemistry people and saying most people think car springs when they hear "suspension".

Look, you didn't know what "monolith" means in a software context. Everyone has stuff they don't know, even in their field. Learn something and move on.

[KronisLV]

> "Monolith" is a term of art in software engineering. You're in a discussion about software engineering.

The problem is that most of these terms are loosely defined and evolve over time. And I do mean in practice, as opposed to some thesaurus definition that gets lost in conversation.

REST? A set of architectural constraints, but nowadays most just selectively pick aspects to implement and forget about things like HATEOAS or hard problems like resource expansion and something like HAL isn't popular.

Microservices? Some think that those need to be "small" which isn't true, and yet somehow we see time and time again people ending with a pattern of service-per-developer instead of service-per-team, because people just aren't good at DDD and aren't experienced with what works and doesn't yet, if they haven't been building systems like that for a while.

Agile? I'm sure that you're aware of what the management and consultation industry did to the term, now we have something like SAFE which goes exactly against what Agile is and shouldn't be allowed to be named after it.

Cloud native apps? Who even knows, everyone keeps talking about them, there are attempts to codify knowledge like "12 Factor Apps" but I've seen plenty of projects where apps are treated just like they were 10 years ago, to the point of dependencies being installed into containers at runtime, logs being written to bind mounted files and configuration coming from bind mounted config files, as well as local filesystem being used for intermediate files, thus making them not scalable.

> Look, you didn't know what "monolith" means in a software context. Everyone has stuff they don't know, even in their field. Learn something and move on.

Another person suggested the distinction between "monolith" applications (referring to the codebase) and "singleton" applications (which describes the concept of only a single instance being workable at a time). That advice is workable and is useful, since now we have two precise concepts to use.

Your advice isn't useful, because while you're right about the "proper" definition, you brush aside the fact that people misuse what "monolith" is supposed to mean and don't engage with the argument that we need a proper set of terms to describe all sorts of applications to avoid this.

If we don't address this, then monolithic apps become some sort of a boogeyman due to claims that they aren't scalable when that isn't true for all monolithic apps (since that should only imply how the code is deployed), just the subset of arguably badly written ones, as explained in my other comment.

So essentially:

  - a single codebase that can only run as a single instance --> could be called "singleton apps"
  - a single codebase that can be deployed with multiple concurrent instances --> the proper definition of "monolith"

Without this, you cannot say something along the lines of: "Please don't make this monolith into a singleton app by using local instance memory for global business state." and concisely explain why certain things are a bad idea.

Hope that clears it up.

[wlonkly]

> - a single codebase that can only run as a single instance

I've heard this referred to as a "singleton app".

[KronisLV]

That's pretty good and sounds like an apt description!

[ajmurmann]

There is no limitation that this has to be served by a single server

[sdrwefgfvb]

Wait, so in what sense is it a monolith then? :D

[nkozyra]

Not sure how much is sincere here, but a monolith typically means "not microservice architecture" and has very little-to-nothing to do with the ops side.

[sdrwefgfvb]

Well if you're distributing requests over multiple servers, I wouldn't call that a monolith.

[xeromal]

You're mistaken then. Monolith is just project organization. Think microservices and then realize the monolith is on the other end of the spectrum. It's in the name.

[foobiekr]

You keep posting this. A monolith is a software architecture. It says nothing about there being one instance of that monolithic process or a billion any more than "microservice architecture" becomes untrue unless you have k instances of any particular service.

It's like saying "this thing says it is built as a suspension bridge, but there are ten of them, so I wouldn't call it a bridge."

[ancount]

Here is some Microsoft documentation describing how a monolith application can be deployed behind a load balancer[0]. HTH

[0] https://docs.microsoft.com/en-us/dotnet/architecture/contain...

[soggybutter]

If your definition of a monolith refers to a singular server, then I've never worked on a monolith in my life. Even the crufty old ASP apps I used to work on weren't monoliths.

This would also mean that deploying a microservice architecture onto a single server, using docker for example, would be considered a monolith. Or even better, any application that leverages a database on a separate server is no longer a monolith! That's way easier than rearchitecting your whole app.

I've never come across a definition for monolith in the context of software that had anything to do with the actual infrastructure employed

[macksd]

Nevertheless, that's a pretty common term when the compute portion of the application is a single tier, but that doesn't mean that the tier can't scale over many stateless compute nodes.

[emaginniss]

I believe you're alone in using that definition

[aqme28]

Monolith refers mostly to the code, not the distribution of app servers. There's one app (a monolith) served multiple times.

[ajmurmann]

You got some good, but short answers. Since you asked me, I'll give you my definition: It means that your app servers are homogenous. They are all running the same server code in similar configuration. How many instances you are running has nothing to do with it being a monolith or not. You need more than one server just to get some fault tolerance. In practice your scalability bottleneck will typically be your database as all servers share that and your servers themselves ideally are stateless (they typically are for Rails apps, as Zendesk is using Rails).

Microservices on the other hand in theory have a number of very small services that at most share some libraries, but can be deployed independently and could each even be written in different languages if you want. This mostly is an advantage if you know that different portions of your app are gonna scale very differently from each other and rely on disparate data models as well and only share a small interface. In practice, things frequently don't end up as clear cut as you expected on the outset and you end up having to heavily coordinate many of your deploys across services.

[hadrien01]

Rails monoliths (any monolith actually) don't have to be deployed only once, they can still scale if needed.

[nkozyra]

A site that big is clearly distributed and parallelized in addition to the caching.