[anoop_kumar]

Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.

Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.

Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.

[yolo-auto]

actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.

but were losing money right now obviously.

we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.

People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.

its been fun though. you should check it out

[shaism]

Which provider do you use to access the MI300x?