[dwa3592]

>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.

What are the LLM standards?

Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.

[Aurornis]

I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.

Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.

It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.

[dwa3592]

Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.

[andrekandre]

the local models open source harnesses are really improving quite fast; just a few months ago i couldn't get any tool calling to work, and responses were very slow (thinking going on too long etc) but now with some newer models on my macbook air, tool calling works and depending on the model, it returns from thinking fairly swiftly....