The sub-discipline of mathematics that deals with this is called "metamathematics". For a start see https://en.wikipedia.org/wiki/Metamathematics