Have you though about using a pseudonym instead of your real identity for this? My contract even forbids me to contribute to OSS. You can’t advertise with any of your OSS work on your CV though.
The standard Google employee contract contains this restriction. It's easier for them to institute a blanket ban on open source contributions (including in one's spare time) and then consider exceptions on a case-by-case basis. They have a dedicated internal committee that fields employee requests to work on open source projects.
What, in your spare time? That seems... restrictive