Hacker News new | ask | show | jobs
by creesch 659 days ago
LOL, are you kidding me? LLMs including the gpt family copilot uses are very good at sneaking in these sorts of thing.
1 comments

Show me 1 example.
There's more than one mistake in the code given, and I tried just once.

"Create Python code for iterating a file with a cache."

    class CachedFileIterator:
        def __init__(self, file_path):
            self.file_path = file_path
            self.cache = []  # List to store cashed lines
            self.current_index = 0  # Index for the current line to read

        def _read_lines_from_file(self):
            """Read lines from the file and cache them."""
            with open(self.file_path, 'r') as openFile:
                for line in f:
                    self.cache.append(line.strip())  # Strip newline characters and save to cache

        def __iter__(self):
            """Return the iterator object itself."""
            return self

        def __next__(self):
            """Return the next line in the iteration."""
            if self.current_index < len(self.cache):
                result = self.cache[self.current_index]
                self.current_index += 1
                return result
            else:
                # If we reach the end of the cache, we read more lines from the file if necessary
                if len(self.cache) == 0:  # Cache is empty, read file
                    self._read_lines_from_file()
                    
                if self.current_index < len(self.cache):
                    result = self.cache[self.current_index]
                    self.current_index += 1
                    return result
                else:
                    # If still no lines, we stop iteration
                    raise StopIteration

    # Example usage:
    if __name__ == "__main__":
        file_path = 'example.txt'  # Replace with your file path
        
        iterator = CacheingFileIterator(file_path)
        for line in iterator:
            print(line)
Garbage code is bad enough, but it's not like people have never had to walk juniors through mistakes before LLMs.

But this is actually so much worse for that same reason - the type of developer who'd submit Copilot output (I can call it that, as it's definitely not code) for a PR is unable to respond to any comment beyond asking Copilot again and wasting everyone's time with 6 more rounds of reviews. I've literally had to write test cases for someone else and told them "You can't ask for another code review until your code passes these."

Bit of a tangent, though related. It looks like you accidentally stumbled into a version of test driven development ;)

With the big difference obviously being that typically the developer who writes the test also will write the code.

In some situations, this actually makes sense to do with junior developers as part of their training. Where a senior developer sits down with them and write out the tests together, then with the tests as a guide they are thrown into the waters to develop the functionality.

Of course, I suspect that in this case, you were not dealing with a junior. Rather the sort of person who looks at your tests, still is confused and asks for a "quick call" to talk about the tests.

> "You can't ask for another code review until your code passes these."

Such a good idea :-) Maybe for job applications too and any at home work sample tests

What do you see as mistakes? I see some weirdness, but the spec is just not complete - there was no requirement for rewinding, multiple users, etc. in the request so it's not implemented.

The only thing I'd call an actual mistake is using an empty list to mean both an empty file and an uninitialised value.

The file object is named "openFile", but used as "f". The class is defined as "CachedFileIterator", but used as "CacheingFileIterator". That's two typos, before discussing the actual code.
Well, there's also the fact that the entire thing could be replaced with...

    def cached_file_iterator(file_path):
        with open(file_path, 'r') as f:
            lines = [ line.strip() for line in f.readlines() ]
        yield from iter(lines)

    # Example usage:
    if __name__ == "__main__":
        file_path = 'example.txt'  # Replace with your file path
        
        iterator = cached_file_iterator(file_path)
        for line in iterator:
            print(line)

Which is functionally identical and FAR less code to maintain.

    for line in f:
is multiple mistakes in a single line.
What are the mistakes there?
f doesn't refer to anything.

iterating over the file object at all instead of just calling self.cache = openFile.readlines() means that calling strip() the line below removes data beyond just the trailing newlines.

The most obvious one:

    with open(self.file_path, 'r') as openFile:
        for line in f:
`f` does not exist. It should be `openFile`.
One is that the variable is called openFile and not f. I don't know enough python to see something else wrong with that but would love to know too, since I've written such a line just last week.
Show me that copilot never makes a mistake or introduces variables that never have been initialized...
This is a strawman, he never said it didn't make mistakes.
Oh for crying out loud, I obviously mean these specific mistakes. If you have worked in any capacity with LLMs like this you would have seen them variables or suddenly switch up the convention of how they're written.

Certainly if you are in a conversation mode after a few back and forths this happens from time to time.

I am just not going to spend my time digging to previous prompts of code I might not want to share just to satisfy a random internet person .

The models I've used don't make typos on variable names that already exist in the context. Typos are not the failure mode, this is literally the easiest text prediction task they can do.
What you guys probably want to do instead is get to a common definition of what a typo is. Personally, I understand it as a typographic error, which is a fancy way of saying a spelling mistake (a mistake on a letter), not a mistake where one use a word for another.

Maybe you meant the latter?

I, for one, have not have this experience with LLM creating new variable names when they already defined one.

Lots of mistakes, but never this one.