One thing that is used a lot in physics are known as Dirac Deltas[0] that is, in very informal terms, the derivative of the function f(x) = 0 for negative x otherwise f(x) = 1.
Physics are very good at working with concepts and abstraction before any formal mathy justifications can be found, but the only way to formaly work with a dirac delta that makes sense formally is defining it in terms of measures
That is not true, it is not the 'only way' to formally deal with them. A better way to think of them is as the vector space dual of functions (/forms) under the pairing given by integration. No measures required. The measure theoretic explanation is very much "fitting delta functions into our existing machinery" rather than any sort of inherent requirement.
Actually an even better way to think of delta functions is just as a geometric object, a point (or line/plane/etc). Which is somewhat related to the measure theoretic version, but much more simple to think about.
Even if accuracy is finite, the fact that, for example, circles aren't polygons is definitely relevant to physics. You might be able to get all of the relevant physics you need without the continuum by working with several disjoint sets of numbers (the rationals, the rationals-multiplied-by-pi, the rationals-multiplied-by-e, etc) but I'm not even sure of that.