[thaumasiotes]

> This is the side effect of underutilized capital and it’s present in many cases.

"Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low-value work. Option value is real value; being prepared for unforeseen but urgent circumstances is a real use.

[efitz]

I think a better description than “underutilized” would be “sunk capex cost” - Google (or any cloud provider) cannot run at 100% customer utilization because then they could neither acquire new customers nor service transitory usage spikes for existing customers. So they stay ahead of predicted demand, which means that they will almost always have excess capacity available.

Cloud providers pay capital costs (CapEx) for servers, GPUs, data centers, employees, etc. Utilization allows them to recoup those costs faster.

Cloud customers pay operational expenses (OpEx) for usage.

So Google generally has excess capacity, and while they would prefer revenue-generating customer usage, they’ve already paid for everything but the electricity, so it’s extremely cheap for them to run their own jobs if the hardware would otherwise be sitting idle.

[immibis]

There is also a mathematical relationship in queuing theory between utilization and average queue length, which all programmers should be told: https://blog.danslimmon.com/2016/08/26/the-most-important-th...

As you run close to 100% utilization, you also run close to infinity waiting times. You don't want that. It might be acceptable for your internal projects (the actual waiting time won't be infinity, and you'll cancel them if it gets too close to infinity) but it's certainly not acceptable for customers.

[thaumasiotes]

There is a genre of game called "time management games" which will hammer this point home if you play them. They're not really considered 'serious' games, so you can find them in places where the audience is basically looking to kill time.

https://www.bigfishgames.com/us/en/games/5941/roads-of-rome/...

The structure of a time management game is:

1. There's a bunch of stuff to do on the map.

2. You have a small number of workers.

3. The way a task gets done is, you click on it, and the next time a worker is available, the worker will start on that task, which occupies the worker for some fixed amount of time until the task is complete.

4. Some tasks can't be queued until you meet a requirement such as completing a predecessor task or having enough resources to pay the costs of the task.

You will learn immediately that having a long queue means flailing helplessly while your workers ignore hair-on-fire urgent tasks in favor of completely unimportant ones that you clicked on while everything seemed relaxed. It's far more important that you have the ability to respond to a change in circumstances than to have all of your workers occupied at all times.

[bombcar]

> You will learn immediately that having a long queue means flailing helplessly while your workers ignore hair-on-fire urgent tasks in favor of completely unimportant ones that you clicked on while everything seemed relaxed.

Ah, sounds like Dwarf Fortress!

[immibis]

I was thinking Oxygen Not Included.

[dekhn]

In practice it's more complicated than this- borg isn't actually a queue, it's a priority-based system with preemption, although people layered queue systems on top. Further, granularity mattered a lot- you could get much more access to compute by asking for smaller slices (fractions of a CPU core, or fraction of a whole TPU cluster). There was a lot of "empty crack filling" at google.

[efitz]

TL/DR: You should think of and use queues like shock absorbers, not sinks. Also you need to monitor them.

Queues are useful to decouple the output of one process to the input of another process, when the processes are not synchronized velocity-wise. Like a shock absorber, they allow both processes to continue at their own paces, and the queue absorbs instantaneous spikes in producer load above the steady state rate of the consumer (side note: if queues are isolated code- and storage-wise from the consumer process, then you can use the queue to prevent disruption in the producer process when you need to take the consumer down for maintenance or whatever).

Running with very small queue lengths is generally fine and generally healthy.

If you have a process that consistently runs with substantial queue lengths, then you have a mismatch between the workloads of the processes they connect - you either need to reduce the load from the producer or increase the throughput of the consumer of the queue.

Very large queues tend to hide the workload mismatch problem, or worse. Often work put into queues is not stored locally on the producer, or is quickly overwritten. So a consumer end problem can result in potential irrevocable loss of everything in the queue, and the larger the queue, the bigger the loss. Another problem with large queues is that if your consumer process is only slightly faster than the producer process, then a large backlog of work in the queue can take a long time to work down, and it's even possible (admission of guilt) to configure systems using such queues such that they cannot recover from a lengthy outage, even if all the work items were stored in the queue.

If you have queues, you need to monitor your queue lengths and alarm when queue lengths start increasing significantly above baseline.

[b112]

I doubt they are doing this, but if they did burn in tests with 3 machines doing identical workloads, they could validate workloads but also test new infra. Unlike customer workloads, it would be OK to retey due to error.

This would be 100% free, as all electricity and "wear and tear" would be required anyhow.

[franga2000]

In the case of compute, you can evict low-priority jobs nearly instantly, so the compute capacity running spot instances and internal side-projets is just as available for unexpected bursts as it would be if sitting idle.

[nathancahill]

Same effect when leasing companies let office space sit unoccupied for years on end. The future value is higher than the marginal value of reducing the price to fill it with a tenant.

[Bjartr]

That may be part of it for spaces properties left unleased for years, but I believe it's not the only part.

I believe the larger factor, and someone correct me if they have a better understanding of this, is that for commercially rented properties the valuation used to determine the mortgage terms you get takes into account what you claim to be able to get from rent. Renting for less than that reduces the valuation and can put you upside down on the mortgage. But the bank will let you defer mortgage payments, effectively taking each month of mortgage duration and moving it from now to after the last month of the mortgage duration, extending the time they earn interest for.

So if no one want to lease the space at that price after a prior lessee leaves for whatever reason, it's better for the property owner financially to leave the space vacant, sometimes for years, until someone willing to pay that price comes along, than to lower the rent and get a tenant.

[bombcar]

This is mostly correct. People assume commercial loan terms are like single-family homes "but larger" but they're not. They basically are all custom financial deals with multiple banks and may be over multiple properties. As long as total vacancy isn't below a cutoff the banks will be happy, but lowering rents "just to get a tenant" can harm the valuation and trigger terms.

Part of the reason things like Halloween Superstores can pop in is the terms often exclude "short term leases" which are under six months.

Also when you're leasing to companies, they are VERY quick to jump at lower prices if available, which means that if you drop the lease for one tenant, the others are sure to follow, sometimes even before lease terms are up.

[b112]

Many cities only tax on leased property, or have very low rates on unleased property.

[khafra]

Land Value Tax would fix this.

[unyttigfjelltol]

Real estate is a playground for irrationally hopeful or stubborn participants.

[axus]

I'm going to say this the next time I argue I need my servers online 24/7.

[thaumasiotes]

I'm not really sure I'm following you.

[bombcar]

Yeah, airlines make "more return on capital" by faster turn-around of planes to a point - if they are utilizing their airframes above 80 or 90 or whatever percent, the airline itself becomes extremely fragile and unable to handle incidents that impact timing.

We saw the same thing with JIT manufacturing during Covid.