[clmnt]

Xet might be one of the most important AI technology that nobody is talking about!

Under the hood, it now powers 5M Xet-enabled AI models & datasets on HF which see hundreds of terabytes of uploads and downloads every single day.

What makes it super powerful is that it massively speeds up & reduces costs of data transfer thanks to methods like content-defined chunking (CDC). Instead of treating a file as an indivisible unit, CDC breaks files down into variable-sized chunks, using the data to define boundaries.

That's what allows Hugging Face to offer a platform for 10 million AI builders in open-source at a fraction of the cost.