| > I suppose if I had a 7 digit budget I could get a better deal. We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. > I'm actually surprised you have 100% inference utilization - customer load typically scales dynamically, so with on-prem servers you would need to over-provision. It is pretty easy to achieve 100% inference utilization if you can find inference work that does not need to be done on-demand. We have a priority queue and the lower priority work gets done during periods with lower demand. > CEOs don't usually order hardware, they have IT people for that, with input from people like me (ML engineers) who could estimate the workloads, future needs, and specific hw requirements (e.g. GPU memory). Judging by this conversation it seems like "people like you" may not be the best people to answer this question since the best hardware quote you could get was at a >100% markup! At a startup that specializes in ML research and work the CEO is going to be intimately familiar with ML workloads, needs, and hardware requirements. > And when your people come to you asking for budget, while you're trying to raise the next round, you're more likely to approve the 'no high upfront cost' option, right? If the break even point is 6-7 months and our runway is longer than 6-7 months why would this matter? |
Now I’m really curious - if you can share - how much did you pay, and when was it? Are you talking about 40GB or 80GB cards? How did you negotiate? Any attempts I made were shut down with simple “no, that’s our final price”. What’s the secret?
At a startup that specializes in ML research and work the CEO is going to be intimately familiar with ML workloads, needs, and hardware requirements.
I work at a startup which builds hardware accelerators, primarily for large NLP models. It’s a large part of my job is to be intimately familiar with ML workloads, needs, and hardware requirements. Our CEO definitely doesn’t have enough of that knowledge to choose the right hardware for our ML team. In fact even most people on our ML team don’t have deep up to date knowledge about GPUs, GPU servers, or GPU server clusters. I happen to know because I always had interest in hardware and I’ve been building GPU clusters since grad school.