[nojvek]

Yasssssss! Thank you.

This is the future. I am predicting Apple will make progress on groq like chipsets built in to their newer devices for hyper fast inference.

LLMs leave a lot to be desired but since they are trained on all publicly available human knowledge they know something no about everything.

My life has been better since I’ve been able to ask all sorts of adhoc questions about “is this healthy? Why healthy?” And it gives me pointers where to look into.

[zitterbewegung]

I actually think Apple has been putting neural engines in everything and might be training something like Llama3 for a very long time. Their conversational Siri is probably being neglected on purpose to replace it . They have released papers on faster inference and released their own models. I think their new Siri will largely use on device inference but with a very different LLM.

Even llama.cpp is performant already on macOS.

[mcculley]

They are not “trained on all publicly available human knowledge”. Go look at the training data sets used. Most human knowledge that has been digitized is not publicly available (e.g., Google Books). These models are not able to get to data sets behind paywalls (e.g., scientific journals).

It will be a huge step forward for humanity when we can run algorithms across all human knowledge. We are far from that.

[neurostimulant]

There is a rumor that OpenAI might've used libgen in their training data.

[mcculley]

Someone will. The potential gains are too high to ignore it.

[nojvek]

We are talking about trillions of tokens.

I’m sure the big players like Google, Meta, OpenAI have used anything and everything they can get their hands on.

Libgen is a wonder of the internet. I’m glad it exists.

[mcculley]

I am also glad that libgen exists. Liberating human knowledge from copyright will improve humanity overall.

But I don’t understand how you can be sure that the big players are using it as a training corpus. Such an effort of questionable legality would be a significant investment of resources. Certainly as the computronium gets cheaper and techniques evolve, bringing it into reach of entities that don’t answer to shareholders and investors, it will happen. What makes you sure that publicly owned companies or OpenAI are training on libgen?

[maxboone]

Groq is not general purpose enough, you'd be stuck with a specific model on your chip.

[tome]

I'm not sure what you mean. The GroqChip is general purpose numerical compute hardware. It has been used for LLMs, and it has also been used for drug discovery and fusion research (https://www.alcf.anl.gov/news/accelerating-ai-inference-high...).

[I work for Groq.]

[tome]

And just in case it's not clear: I'm saying Groq can be used for arbitrary AI inference workloads, and we aim to be the fastest for all of them. We're not tuned to any specific model.