Thorium is not much radioactive indeed, but radon it emits is. That's why f.ex. building houses on a granit (that is generally thorium rich) without proper ventilation underneath can result in serious health issues.
There's loads of cities built on and with granite, and long before any awareness of Radon came about, without undue health effects. Scotland particularly - Aberdeen is even nicknamed the Granite City. Modern levels of draught exclusion can make for a little more of a problem of course.
You can map a specific radon level to probability of getting lung cancer. At high levels, it's equivalent to smoking cigarettes, and for smokers, it increases your cancer risk 7-8x.
At the levels where it is recommended that you remediate your home, your likelihood of dying from lung cancer is equivalent to dying in a car accident. That's potentially 50 people a year when you look at a population of 200,000 people.