This is very important. Many students hit a wall when physics passes beyond lay intuition and whatever may they already know well. Advancing in mathematics is essential to feeling comfortable in advanced physics.
I discovered this last year after arrogantly jumping into the first volume of Feynmans Lectures on Physics.
50 pages in, I decided to take a step back and read a calculus book first, but wait my algebra and trig are crap so back to the basics. So yesterday I hit LCM and GCD applications and factoring which are very basic. So, I'll probably resume the initial book in a couple of years or so...
Math is wide and deep. You won’t need to cover every topic in math to get going with physics. If you really are interested in physics there are many things in math, which are, well, less important (for doing basic physics). For example LCM, GCD and factoring. I guess, these things are somewhat important in Computer Science, but I never encountered them in a physics problem.
So to get started with physics, I would suggest that you focus mainly on analysis (differentiation and integration) and vector algebra. As an addition maybe the basics of complex numbers. This can be learned relatively quickly.
With these you should be able to follow the Feynman lectures or watch the very fine „Theoretical Minimum“ series by Susskind (http://theoreticalminimum.com)
I'm doing a full review of mathematics at the moment. Not in depth, more of a "here's an application of the GCD function" so I know what tools to use to solve specific problems. All this is beneficial for the day job as well who expect to see some value from my time spent even though I'm not being totally honest with the objective to them. Realistically I want to think abstractly in the terms of mathematics and develop some intuition.
Was completely unaware of the Theoretical Minimum series. Thanks for that.
Edit: I'm reading Mathematics: From the birth of numbers by Jan Gullberg as a text. Wonderful book. Covers just about everything and is beautifully written by a non mathematician with no assumptions spared and no education target. In fact the forward is mainly bitching about the education system. Slightly worried I will get distracted by this book but that's never a loss!
I do not know "Mathematics: From the birth of numbers" but judging from the Amazon quick view it seems to cover a lot of ground (BTW: one thing I missed in my list are the basics of differential equations).
Over 1000 pages is quite a long read, though. I never managed to read a (science) book as big as that from cover to cover myself. One thing I learned through the years is to never use only one book for learning. Books have different styles and not every style fits to every student. Additionally one book might be good at one specific topic and weak on another. So nowadays I always use a couple of books (or online resources) to learn a new topic.
I have a number of other books as well that I use as a reference as well so no problems there (calculus for the practical man has some different insights). Oh and betterexplained.com.
On a related note, Mary Boas's text, Mathematical Methods in the Physical Sciences does a great job of giving you the necessary bag of tricks to learn all of undergraduate level physics (and probably much more) without diving too deep into any single topic. It should be sufficient to give you lots of intuition until you decide to pursue something at much greater depth (although doing that alone, and without a professor/PI/expert of some sort is realistically, almost definitely a waste of effort).
One suggestion: make sure you don't just read about math, but also try solving exercises and problem. These are very important for actually learning the material.
50 pages in, I decided to take a step back and read a calculus book first, but wait my algebra and trig are crap so back to the basics. So yesterday I hit LCM and GCD applications and factoring which are very basic. So, I'll probably resume the initial book in a couple of years or so...