You do if you want Google to send any traffic to it. Your only option is to bend the knee unless you're site is nothing more than a blog which you do for fun.
All that AMP does for speed is impose limits on CSS and JS size (plus givs access to Google CDNs). You can simply write small assets without the need for Google to specify this.
I've recently built an AMP version of a marketing website (since AdWords supports AMP cache, thought it would be useful for speed). The truth is, while AMP as a library does provide solid speed improvements on its own by implementing all the best practices, the real speed improvements is the cache with preloading - as well as the other stuff it does like image optimisation.