[biot]

It would be interesting to see how you could have resolved the issues. I think eliminating the search index generation on instance startup would have solved almost all the issues you were experiencing. For comparison, Khan Academy serves up 6 million active users a month:

http://highscalability.com/blog/2013/4/1/khan-academy-checkb...

That's likely many orders of magnitude more than the traffic levels you're experiencing. Clearly the platform works, but you need to architect your application to work with it rather than trying to shoehorn App Engine to work with your architecture.

If you don't have the ability to do that (due to time pressures or other factors), then you made the right call to move to a platform you are familiar with.