This is definitely worthy of concern. There's an infamous case where an AI was trained to detect cancer from imaging but all the positive examples included a ruler (to measure the tumor) so it turned out it just was good at detecting rulers.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9674813/#:~:tex....