Hacker News new | ask | show | jobs
by whatever1 3 days ago
Why a retailer needs 40k servers? What are they serving? To whom?

What is wrong with system designers these days? Are they designing or just selling?

7 comments

They've got a little over 5,000 stores. Probably a few different offices. Each store probably has several VMs or so so they can continue to run isolated if their connectivity to the mothership disappears briefly. You'll probably want the basics of running some kind of local auth service (maybe AD, maybe something else), your system running the POS platform which might even be an app server and a database server as separate VMs, probably some VM running various building management stuff, a VM to run the security camera platform, I dunno what else.

That's just for the operations of the local stores and we're now at probably >25,000 VMs and we've only touched the retail locations. We still haven't addressed logistics locations, corporate offices, stuff to manage their customer-facing applications and websites, etc.

When I first saw 40,000 VMs I too thought it was a bit excessive, but when you're an org wit several thousand locations that you want to be somewhat self-sufficient things add up quickly!

Online ordering. Backend logistics (like icebergs, most of a supermarket is invisible). Stocktaking. Financials. They've probably got several role-isolated servers per store, each with a backup.
Is it that hard to imagine? They do 100B USD / yr revenue as a supermarket chain with 330k employees and a massive logistics operation. The software supporting the whole shebang is not gonna run on a spare macbook pro in a cupboard.
You motivated why they need a big database. Still unclear to me why they need 40k VMs and not just 100.

Its ok for Amazon to do it since they paid for the physical machines anyway and they want to dogfood their AWS services, it does not make sense for someone who rents compute and licenses.

They have over 3000 stores. Maybe each one has a handful of VMs.

Maybe each self-service checkout runs a VM, or each staffed checkout.

They wouldn't want to rely entirely on a remote database to be able to sell products.

And their Dunnhumby division is a huge data consultancy in its own right, and likely the largest user of compute within the wider Tesco org.
How do you imagine cash registers work these days? With 5,000 stores worldwide, 40k servers is 8 computers per store, which doesn't seem excessive to me.
5000 stores, let's say 10 checkout lines per store, just to overprovision, so at worst 50000 simultaneous transactions going on, but probably way less. You can do that with a single server, but you'd want some redundancy and spare capacity.

I worked with a Danish retailer with +3000 store in ~50 countries, and even adding their webshop on top and they were closer to 200 (maybe 300) servers (most VMs). Then you need the ActiveDirectory, office IT, all that stuff, with redundancy and it adds up quickly... but not to 40K.

What I will say that people forget is that production might be 8 beefy VMs, you still need to replicate that to a number of test systems, staging environments and so on. So a 8 node production cluster because maybe 24 servers when accounting those other environments.

I'd assume most of the big supermarkets have a 4-5 host cluster with the small local stores having a 2-3 host cluster. You've got the software to run the tills sure, but also the loyalty card system (which seems to have a local cache at each site based on how quickly it returns your first name), VoIP, Door Access systems, BMS, Digital Signage, Scan as you shop systems, CCTV, Stock management systems..

I can see how you hit 40k pretty quickly.

See these numbers I buy. 40k smells over engineering to me (or more likely a 3P salesman made a huge bonus after selling the architecture to tesco)
I imagine that cash registers have the cpu power of at least a cellphone and that they can store transactions over internet in the central company database.
at Tesco scale you really dont want a central database even if its big and high availability. if one store loses connection because of isp issues it would shut down everything and make it impossible to serve customers in that location. a global outage would cost billions. latency is also a big deal when you got a crowd lining up in front of every checkout lane and the self serve machines, at that point it can even turn into a safety/liability issue.

and dont forget that different countries have different rules about customer private data and payment information. if you send eu customer info into the uk for processing you might be breaking privacy laws. some places do automatic tax reporting where you need to send info to a country specific tax office api, get back a code and print it on the receipt.

cash registers dont work alone. its connected with inventory management, employee perf monitoring, payment processing and other things that make more sense as a store local service sending regular reports to corporate instead of waiting for round trips on every operation.

8 servers per store does sound excessive unless of course they are running vm ware on all the computers in the stores, and they count as "servers".
Lots of these machines are dynamically created and provisioned depending on load factors nowadays.

The days of manually setting up servers in hyperscaling-environments are long long gone.

Example: Your GitLab CICD needs Runners. They are dynamically requested "somewhere in our cloud somwhere in the world" and then spun up and configures fully automatically. No human touches this stuff anymore.

"No human touches this stuff anymore." And then Clown Strike happens ... amplified by Opus and "agile" BS.
To the business? logistics, facilities (think power, HVAC, refrigeration systems), security, HR, finance, accounting, legal, purchasing, sales, marketing, etc. Probably with multiple environments (dev/qa/prod/DR) for anything important.
If you can't imagine the scale of a retailer's etnerprise architecture, maybe it's a you problem?