[M4v3R]

Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.

[swyx]

so why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption