[justinph]

With the upcoming Core Web Vitals, you'll get penalized for this. It measures the aggregate speed of chrome users.

[djitz]

This is already a prominent section of Google's search console and they fire off warnings with with the perceived issues which you validate after fixing, etc. I'm sure it will become a major factor very soon.