That I can agree with. I don't often ride long distance rail because even on the Northeast line which is I think the line with the highest ridership you often have to stop for freight trains and other trains.
I've never had to stop for a freight train on the northeast (between DC and NYC anyway), because Amtrak owns those tracks. You do sometimes have to stop for local commuter trains that share the tracks, or (more commonly) an acela overtaking a regional. By far the most common reason for delay, though, is the aging equipment.