[vjk800]

My experience has been that AI is much more useful on my own systems than on company systems. For AI to (currently) be useful, I need to choose my own tooling and LLM models to support AI centered workflow. At work, I have to use whatever (usually Microsoft) tools my company has chosen to purchase and approve for my corporate computer, and usually nothing works as well as on my own machine where I get to set it up as I want.

[sigmoid10]

What you are describing is a failure to integrate AI into said company systems. I have seen quite a few companies now that buy MS AI products with great hopes only to be severely disappointed, because they may as well have just used vanilla ChatGPT (in fact then they would at least get newer models faster). But there are counter examples too. If you can pull all your company documentation into a vector db and build a RAG based assistant, you can potentially save countless hours across your workforce and possibly customers too. But this is not easy and also requires some level of UI interactivity that noone really offers right now. In fact they can't offer it, because you usually need to integrate ancient, arcane sources into your system. So you do have to write a lot of integration code yourself at every step. Not many companies are willing to spend that kind of money and effort, because managers just want to buy a MS product and be done with improving efficiency by next quarter.

[mark_l_watson]

I have been using vector based RAG for about two years now, I am not knocking the tech, but last year I started experimenting with going way back in time and also in parallel trying BM25 search (or hybrid BM25 and vector). So: not even a very good example use case of LLMs, the tech is not always applicable.

EDIT: I am on a mobile device and don’t have a reference handy but there have been good papers on RAG scaling issues - basically the embedding space gets saturated (too many document chunks cluster in small areas of the embedding space), if my memory is correct.

[sigmoid10]

Depends on your use case. A system that can do full text and semantic search across a vast archive, open files based on that search to retrieve detail and generate an answer after sifting through hundreds of pages is pretty powerful. Especially if you mange to pair it with document link generation and page citation.

[mark_l_watson]

I agree with you, but then I am retired so my opinion is not that relevant.

I have a sweet spot for using just Emacs, no other IDEs except very occasional use of AntiGravity. For a particular fun project or researching how to use generative models in applications, I like to start low: see if a small local model with appropriate tool use or agentic library will get the job done, if not move up to using something cheap like gemini-3-flash, and only if none of these approaches work, then use an expensive model.

I was advising a friend’s company last month on their application that effectively uses LLM models, but I was blown away by their zeal to spend lots of money on tokens.