That's why it's nice of Let's Encrypt to publish their current and backup intermediate CAs (X3 and X4) https://letsencrypt.org/certificates/. That's what I pin for my sites.
It's worth noting that Let's Encrypt makes no guarantees regarding the ability to sign under either key[1]. Both keys could be rotated without prior notice (though that would admittedly be unlikely unless there's an emergency).
For example, some kinds of HSM bugs might conceivably lead to issuing a new intermediate after an HSM software update (like if an HSM vendor said that it had concluded that its CSPRNG was seeded inadequately with fewer effective bits of entropy than the specifications required, or something).
In support of being cautious about predicting what intermediate will be used for issuance at a given time, there was also once a Let's Encrypt Authority X2 (also browser-trusted by virtue of being signed by IdenTrust's root), but issuance skipped directly from the X1 to the X3 intermediate in March 2016.
[1]: https://community.letsencrypt.org/t/official-hpkp-support-fr...