[SatvikBeri]

This is a fairly interesting philosophical question. Personally I view (and use) Math as a set of abstractions to understand the world, and use Mathematical Logic to help make Math more rigorous and proofs more checkable. But if it turned out that Mathematical Logic had some flaws as it is formalized today, I wouldn't throw out Probability Theory or Algebra. Instead I would seek a formalization of Mathematical Logic that made those things useful.

I don't think that there's any formal system you can use as the basis of all Math, though. For example, ZFC can't talk about proper classes, but we'd like to be able to make statements about the class of all sets, and the collection of all classes, etc.