Some of the simple tools might not be using ffmpeg per se, but using the libav or similar libraries. ffmpeg is just a tool built to utilize the functionality of multiple libraries like this.
No, ffmpeg itself is a pretty small wrapper around them. But the rest are true libraries and not frameworks, so you have to make all kinds of playback decisions yourself. A/V sync is hard.