[sexy_panda]

I wonder if this could be done on a FPGA..

[vasco]

FPGA for development, AWS can afford to order custom ICs.

[toomuchtodo]

Could surplus crypto miner GPUs be repurposed for this?

[vasco]

No

[Frost1x]

With enough gates you can do anything on an FPGA

[pclmulqdq]

How much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.

[tgsovlerkhgsel]

Probably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high).

I would expect this to become part of newer generations of CPUs once it becomes popular.

[wmf]

Intel sells a gzip PCIe card called QAT. Not many seem to be sold.

[NelsonMinar]

I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.

[sexy_panda]

It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.

My Intel i9 11900k seems to support it or has it built-in.

[pclmulqdq]

QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.

[zxcvbn4038]

It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.