But there are lots of trains leaving at any given time going to different places. There's no way to know which train somebody is heading to without asking them.
Yes there are. That's the whole problem. It's complete chaos. There are several hundred people huddled in one area looking at one giant screen and there are several trains being called in sequence. How would you know that a given person was waiting for the 4:30pm Acela to Union Station to be called and not the 4:33pm Northeast Regional to Boston?
As somebody who commuted for a long time at peak rush hour back and forth from Penn Station to D.C., I can tell you that following people who "seem to know where they're going" would not be a reliable strategy.