Depending on what you eat you could still be well below the average environmental impact of a resident of an industrial nation. If you're vegan/vegetarian then 4000 calories a day from beans would be far less impactful than a 2000 calorie diet of meat.
Fit people need to eat more. Muscle takes calories to maintain. You can't maintain muscles without exercise. Your body expects that you're running down prey on a periodic basis. If you don't meet those expectations not everything will work as it should.
Humans evolved to move around a lot. Avoiding exercise to save food will lead to a host of ailments. Sarcopenia is a hidden epidemic among older people in developed countries.
Fit people need to eat more. Muscle takes calories to maintain. You can't maintain muscles without exercise. Your body expects that you're running down prey on a periodic basis. If you don't meet those expectations not everything will work as it should.