Did you consider using any existing any canvas drawing libraries like Fabric.js? Having recently built an app around Fabric it’s nice to see some alternatives.
If you're looking for alternative canvas libraries, you can find a comparison test of some of them here[1]. Links to the libraries can be found in the GitHub linked to from that page.
[1] - https://benchmarks.slaylines.io/