[chiphuyen]

Hi, I'm Chip, the author of the post. I spent waaaay too much time doing research for this. The AI engineering layer was especially hard because so many tools have similar and/or overlapping features. It was also a lot of pain trying to understand the repos that only have Chinese in the README do.

The full list of the repos is published here: https://huyenchip.com/llama-police

[swyx]

great list! as someone who's also trying to map the ai engineering landscape... i wonder what u think of adding other parts of the AI stack (https://www.latent.space/p/dec-2023). right now you have 4 categories and those are all in the text/code-heavy RAG/Agent world, but i think the space has broadened out a bit as i see it. for example, you could add:

- finetuning/other post-pretrain model tools (axolotl, mergekit <- all made and used by people without traditional ML engineer/researcher background)

- multimodal models/frameworks like vocode and comfyui

- AI UX tools like vercel ai sdk

- synthetic data generation tooling? whatever the nous pple have made

open question whether inference frameworks like llama.cpp/ollama or vllm and tgi count as AI Eng tools? again given the background of ggeranov and the students behind the other projects, arguably yes but ofc it starts to bleed into classical mlops here. (update: i see u have them in the "model development" category, ok fair)

[chiphuyen]

IMO, the classical mlops is closer to the genai stack than most people think. E.g. experiment tracking is the same: with classical mlops, you experiment with hyperparams, with genai, you experiment with prompts. Similarly, finetuning is just an extension of training. Even vector databases for RAG is just vector search + databases, both of which have been around forever.

The post-train world is what I find to be the most fun. Techniques like model merging, constrained sampling, and all the new creative techniques for inference optimization and faster decoding are super cool!

[braza]

Hey Chip, thanks for your contributions along the years and the amazing book.

2 questions: From what you researched, how many of those solutions are ready for production? and Regarding this mortality, what are the let's say top 5 things someone needs to think even before to do a PoC over those tools?

[chiphuyen]

Production is a spectrum. Many of the repos I see are still demowares, but at the same time, most companies I've seen are also still at the PoC phase instead of massively scaling up their GenAI use cases.

I don't think the considerations for adopting a tool has changed. It starts from what problem you want to solve, the money/time budget you have for the solution, ROI of each solution.

I know it sounds generic, but without more detail, it's hard to give a more concrete answer!

[shnkr]

cool. this has been my problem all along. thanks for the time you put in.