It's not an assumption. If you have a railcard, you cannot associate it with a contactless card. That's just not a supported task. Therefore if you travel off-peak, you will be charged more than you would if you had used an Oyster card because you will be charged the public rate instead of the discounted fare you were actually entitled to pay. It's very simple.
https://tfl.gov.uk/fares/free-and-discounted-travel/national...