Hmm...that doesn't sound like normal behavior; my uneducated guess would be that there's a specific webpage or extension with a leak. Next time it happens, it'd be fantastic if you could go to about:memory, save the reports, and file a bug with them attached:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox