[jcranmer]

I recently tried using Claude to generate a lexer and parser for a language i was designing. As part of its first attempt, this was the code to parse a float literal:

  fn read_float_literal(&mut self) -> &'a str {
    let start = self.pos;
    while let Some(ch) = self.peek_char() {
      if ch.is_ascii_alphanumeric() || ch == '.' || ch == '+' || ch == '-' {
        self.advance_char();
      } else {
        break;
      }
    }
    &self.source[start..self.pos]
  }

Admittedly, I do have a very idiosyncratic definition of floating-point literal for my language (I have a variety of syntaxes for NaNs with payloads), but... that is not a usable definition of float literal.

At the end of the day, I threw out all of the code the AI generated and wrote it myself, because the AI struggled to produce code that was functional to spec, much less code that would allow me to easily extend it to other kinds of future operators that I knew I would need in the future.

[dboreham]

I had a somewhat experience with Claude coding an Occam parser but I just let it do it's thing and once I had presented it with a suitable suite of test source code, it course corrected, refactored and ended up with a reasonable solution. The journey was a bit different to an experienced human developer but the results much the same and perhaps 100X cheaper.

[jcranmer]

Some of the issues are undoubtedly that I have a decidedly non-standard architecture for my system that the AI refuses to acknowledge--it hallucinated things like integers, which isn't a part of my system, simply because what I have looks almost like a standard example expression grammar so clearly I must have all of the standard example expression grammar things. (This is a pretty common failure mode I've noticed in AI-based systems--when the thing you're looking for is very similar to a very notable, popular thing, AI systems tend to assume you mean the latter as opposed to the former.)