Miners actually have a lot less power than people realize. Anyone can run their own fully-validating Bitcoin node. Miners do indeed mine the blocks, but _all_ nodes perform verification. If miners try to mine invalid blocks, Bitcoin nodes will simply reject them.
What do you mean by "invalid"? My impression is that if you control 51% of the mining power and have at least some amount of control over the physical network path to your victim (e.g., ability to inject RST packets), you can execute double-spend attacks.
Are double-spend attacks not a concern?
If they are not a concern, why do we need so much computational capacity for mining?