[IgorPartola]

Isn’t the solution to this particular problem to do what a human would do? I just tried this:

    Write me a short Python program that will count how many times a word occurs in a given text.
    
    Sure, I can help you with that! Here's a short Python program that counts how many times a word occurs in a given text:
    
    ```python
    def count_words(text, word):
        words = > text.lower().split()
        count = 0
        for w in words:
            if w == word.lower():
                count += 1
        return count
    
    text = "Snapchat is a fun way to share moments with friends!"
    word = "Snapchat"

    result = count_words(text, word)
    print(f"The word '{word}' occurs {result} times in the text.")
    ```
    
    You can replace the `text` variable with your own text and the `word` variable with the word you want to count. Let me know if you have any questions!

Seems to me that if you give an LLM an environment to run said program it would be able to automatically do this with the correct prompt. This doesn’t solve the insurance policy problem at all but the solution to these problems is different in my opinion.

[araghuvanshi]

That's true, but the problem of long context understanding (say, "summarize each of the situations where the word 'wizard' is mentioned") remains. And that gets much closer to the insurance policy thing.

[IgorPartola]

“Write me a Python program that extracts context surrounding a word from a long text, then creates a prompt to summarize the context.” Still different than the insurance policy problem.

[araghuvanshi]

How much context? One sentence? Two? One paragraph? One page? It's very similar to the insurance policy problem - the text surrounding the information you're looking for, which could be surrounding it by one sentence or 10 pages, is just as important as the information itself

[IgorPartola]

I mean basically this is the well known problem with LLMs: they know how to mince words but don’t understand meaning. Again, I think you didn’t present a good simple example. As presented, the Harry Potter problem is just using the wrong tool for the job and isn’t the same as the insurance policy problem.

But at the end of the day an LLM is right 80% of the time while being 100% confident 100% of the time that it gets the right answer. You can increase that 80% but I don’t see how the current breed of LLMs can learn to self doubt enough to keep trying to understand better.

[Scarblac]

At this point, why get the ai involved? The script is trivial and you still need enough knowledge to judge whether it will work, and to run it.

Why doesn't the AI itself decide that writing a Python script is a good way to approach the problem?