[bennywild]

I'm going to assume "Other than x has a bigger market" can be extrapolated to be "Other than x has more of an upside."

And from that I'll extrapolate that you are in the entrepreneur game to learn and grow as an individual as well as provide something new and interesting for its own sake.

So I'd say "Pick the one that would likely cascade to greater discoveries down the road." In this case, a new mathematics would be a better choice than a new engine based on understood mathematics.