Those can take a big chunk out of the appreciation when you sell and buy, and could make a difference of not breaking even for an additional year or two.
You should because that can be a big factor in comparing returns between home ownership and equities. You can keep $250k of capital gains on your home tax free, while the lowest capital gains rate you’ll pay when selling equities is 15%. That can be a significant difference in what you take home.