|
|
|
|
|
|
by Ancapistani
1115 days ago
|
|
|
FYI: I’ve read the whole comment chain under this post before replying > I told it that it was a senior engineer and I was a random client and asked the same thing. I’m aware that ChatGPT is capable of performing the task I chose. I agree that it’s a relatively weak example, especially if it was my goal to show that AI can’t do this. That’s not really what I was trying to show, though. My point is that you knew two things that many/most people probably wouldn’t: 1) That crafting your prompts to get the LLM to adopt a “persona” impacts the results significantly 2) That monkey patching has downsides, and that understanding when it is appropriate is important I strongly suspect that you knew both of those things because you are a senior engineer yourself. My point is that some level of domain knowledge is necessary to get good results from it; my fear is that one day in the not too distant future, far fewer people will be around with that knowledge, because we’ve slowed the “senior engineer pipeline” down to a trickle through the use of these tools. |
|
|
Here's a relevant paragraph
> That said, it's crucial to understand that monkeypatching can make your tests harder to understand and maintain if used excessively or improperly. When you monkeypatch, you're essentially altering the normal behavior of your code. This can lead to situations where tests pass because of the specific setup in the test and not because the actual code is correct, making it less effective at catching real bugs.
More than that, I could instead tell it at the start to tell me the questions I should ask as followups, here's the list when I told it I think I need to use monkeypatching:
1. When is it appropriate to use monkeypatching and when should I avoid it?
2. What are some common pitfalls or mistakes to avoid when using monkeypatching?
3. Can you give me more examples of how to properly use monkeypatching?
4. Are there any other techniques or tools for mocking or stubbing that I should consider?
I could then just say "answer those". No prompting about assuming downsides or problems.
I agree on point one but it's a very easy lesson to learn - try telling it that it's a pirate. It's also not actually required because someone like me can create the persona prompting and then everyone else can just use it. I'm not subtly crafting a weird prompt, it's just telling it who to be for the most part.
> my fear is that one day in the not too distant future, far fewer people will be around with that knowledge, because we’ve slowed the “senior engineer pipeline” down to a trickle through the use of these tools.
On the other side, what we have here is potentially an infinitely patient, always available mentor. You don't have to crash through mistakes on your own to learn, though some lessons are learned harder because of that.