[mercer]

I've been using phi4 a lot, in most cases to process incoming transcripts of recordings. based on keywords or keyphrases, these transcripts get sent with a custom prompt, or the prompt is in the transcript itself.

I'd say it's suprisingly good at exacting reminders and todos from natural language into json 'action objects', or turning what is essentially a run-on sentence of a transcript into a markdown formatted text.

What I've found most fun to play with is to get it to extract metadata like an 'anxiety score' and tags.

Overall, it's clearly 'dumber' than the hosted big models, and in my case I have to deal with a small context window. In general my 'vibe' is that I have to be clearer and more explicit, and it's usually better to just do multiple passes over the same text with very targeted questions.

Oh, and in my case I definitely notice my laptop screeching to a halt when it's processing a big transcript, but in my case I can specifically delay those jobs to a time where I'm not at my computer.

[smjburton]

Nice, that's a cool use case. I like that the local model gives you more privacy when sharing potentially sensitive data like voice recordings with an LLM. I've been interested in hosting one locally, but I was curious what I would be giving up compared to the commercial models. It sounds like its still possible to get a reasonable result with some caveats. Thanks.