[xtracto]

I have a very clear example of Gemini getting it wrong:

For a code like this, it keeps changing processing_class=tokenizer to "tokenizer=tokenizer", even though the parameter was renamed and even after adding the all caps comment.

    #Set up the SFTTrainer
    print("Setting up SFTTrainer...")
    trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    args=sft_config,
    processing_class=tokenizer, # DO NOT CHANGE. THIS IS NOW THE CORRECT PROPERTY NAME
    )
    print("SFTTrainer ready.")

I haven't tried with this latest version, but the 05-06 pro still did it wrong.

[diggan]

Do you have in the system prompt to actually not edit lines that has comments about not editing them? Had that happen to me too, that code comments been ignored, and adding instructions about actually following code comments helped for that. But different models so YMMV.