Fascinating! I know about that story from the point at which the search for rigorous foundations had already begun (i.e. late 19th century), but no awareness of the connection to Fourier. Thanks.
Have you read The Mathematical Experience? Some of its essays address Fourier specifically and what it meant for math. The whole book is fascinating, even if you think you know the material.