If that's what you need, it would make all sense to redo the instruction fine-tuning of the model, instead of fiddling with prompt or processing to work around the model settings that go counter to what you want.
At the very beginning of my journey I did some fine tuning with Lora on a (I believe) Falcon model, but I haven't looked at it since. My impression was that injecting knowledge via fine tuning doesn't work, but tweaking behavior does. So your answer makes much sense to me. Thanks for bringing that up! I will definitively try that out.