At least modern ICBMs do a star sight to calibrate at the top of their trajectory but, yes, that’s what inertial guidance is. Draper Labs basically pioneered.
Ground-based launch sites can supply both position and orientation very accurately (few seconds of angle), more accurately than stellar correction can, and their gyro platform can keep it within a few seconds of angle as well.
Submarines can determine their position accurately enough, but their orientation data can be improved upon using stars.
The MIRV bus takes in the angular fix just before it starts giving the warheads their individual nudges.