There are other posts in the series of the author. He was the co-author of BERT! Yet his paper was scoffed at as "just engineering". He knows what he is talking about.
Omg, I was not a BERT coauthor! But thank you so much for writing this, I had no idea that other post could have accidentally implied this. I will revise that section.