The video clock (I assume you mean pixel clock) isn't relevant. The synchronization of video and audio happens on a per-frame basis.
And since the frames are clocked out on the audio-clock, there cannot be any drift.
Having said that, the CGA card does not actually have its own clock. It takes the NTSC base frequency of 14.31818 MHz from the OSC-pin of the ISA bus.
On early PCs, such as the IBM 5150/5160, the CPU frequency is also derived from this same base clock (which is why you get the oddly specific speed of 4.77 MHz).