The cost of the home is normally quoted in dollars today (what the seller receives), not the total dollars you will pay on your mortgage. So interest rates don't have much to do with affordability.
The typical early stage buyer is financing their home and very often buying within 25% of the mortgage they can qualify for, especially in competitive markets. Mortgage rates absolutely drive this affordability.
When you’re bidding against other buyers, the more they can borrow for a constant monthly payment, the higher the equilibrium price will be.
The majority of people base their housing budget on the monthly payment they can afford.
When interest rates are low, people can afford higher principal values.
When interest rates go up, the amount of a monthly budget that can go toward principal goes down.
Housing prices will tend to fluctuate to match what people can afford, and what people can afford is determined by their monthly payment.