[blackbear_]

The procedure you are describing could absolutely lead to overfitting, but you wouldn't know for sure until you test the prompt on an independent dataset.