Turbochargers are pretty light weight by comparison. The energy they store is minimal - they're just there to shunt more air into the engine, not provide 20kW on demand.
Turbocharger fans/impellers are not offset weights though but perfectly balanced and they are also lightweight.
A flywheel on the other hand is intentionally unbalanced, an offset weight, kind of like a clock pendulum spinning really fast, making it incredibly dangerous when something does go wrong. It basically becomes a centrifugal slingshot.