But cameras aren't perfect pinholes though. The center of the sensor and the optical axis of the lens are already not perfectly aligned, and the sensor is not necessarily perpendicular to the lens barrel. These distortions might be larger than any change of size due to an object being in the periphery of the field of view, especially for longer focal length where the rays are more parallel.
Correct. So, in theory it works and in practice it works with limitations. You could probably create a lens/camera calibration profile that could take the practical use further.