[ryanSrich]

Wow. I just read that.

> We operate our own hardware, run our own datacenters, and manage our own network infrastructure.

This seems insane to me. Although I don't work with image processing beyond "saving for web" in Photoshop, so I could be wrong. Why would they not use AWS or any number of other cloud providers where capacity planning is handled for you?

[billyhoffman]

The bigger question is, why not use elastic infrastructure?

Sure, image optimization is CPU intensive, but their use case is super bursty. When the source image changes, you do multiple optimizations (convert to WebP, lossless JPEG, lossy JPEG quality change, etc), cache the results and you are done. The ratio of "optimizing image" to "serving optimized copy" must be insane.

Dedicated physical hardware feels like a waste.

[skuhn]

[I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

> why not use elastic infrastructure?

This would have been beneficial (to a certain extent) to solve the over-capacity issue that we faced, but it wouldn't actually be of that much help in an under-capacity scenario. None of our machines are ever idle -- it's simply a matter of how long the work queue gets.

> The ratio of "optimizing image" to "serving optimized copy" must be insane

Yeah, but the serving an already rendered image doesn't traverse our image rendering stack, as you might imagine. That content is cached at the CDN edge (or failing that, within our rendered output cache). We don't re-do work when we don't have to, that's an inherent part of how we keep the cost of operations right and provide our customers with a service of this complexity at this price point.

[detaro]

Running the delivery/CDN part on AWS would probably be way to expensive, and if you are running parts of your own infrastructure anyways (and thus are already paying for sysadmins, data center, ...) you can put at least the base load of your processing there also. But they could of course have the option to fall-back/scale to a cloud provider, if they didn't "require" Mac OS X...

[Exuma]

I would guess because of costs. At one point I was going to do a video processing app and after doing some calculations the costs were insane, and the savings with actual hardware were tremendous. I imagine image processing is a lot less than video, but maybe the same deal. Then again... they did rack mac pros so who the heck knows what they're thinking.

[skuhn]

[I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]

The main reasons are twofold:

A) cost. I haven't done the math yet in 2017, but my recollection of the difference circa 2015-2016 between GPU instances on AWS and our solution is a COGS that's about 3x higher on AWS. GPU instance prices have come down a bit since then, but they're still very expensive and somewhat supply constrained as well.

B) Technical frameworks. We'll reach an inflection point this year or next where we've added or changed so much of the rendering pipeline that we've basically written it from scratch, but until then we're still reaping the benefit of building on top of CoreImage. It really benefitted us early on in getting a MVP faster, and it's continued to pay (slightly reduced) dividends over the course of the company's life. At some point the dividends will stop, and then we'll move on.

I understand the skepticism around Mac Pros. There are challenges there -- it isn't my most favorite solution of all time. It is a practical solution for us though. I have no room for computer religion or deciding to do things because they're cool or shiny or new. Anyone that's ever spoken to me for a few minutes in person about technology stuff can attest to that.

The Macs don't have IPMI, for example. That sucks but we do have power outlet control and a network installer as a way to back into "out of band management" for them. They're largely stateless, so it's a tolerable solution.

They do run a Unix-like OS (thank god), and they do represent a good price / point ratio for image rendering hardware. We could do a little bit better with Linux servers and GPUs -- but the upside is only around 10-20%, and there's still engineering work to do there. It's on our roadmap to explore this more fully and maybe start taking more concrete steps in that direction. For now, OSX still gives us more than it takes in terms of cost.

[JimmyAustin]

If I remember correctly, a ton of their original image processing code was written for OS X and it was nontrivial to port to another platform (Might have used CoreImage?), so mounting Mac Pros was the only solution they could find.