I would think fonts specifically would have a large negative impact that would affect first contentful paint given that tons of sites load a few common fonts.
Why does Chrome (or maybe other browsers) not just keep a local cache of the most used Google web fonts by default? Seems like an easy performance win if they're used on lots of sites…