"The behavior of the Scene Camera's FoV, probably to keep the subject from going out of frame, right, is also correcting the camera position, along with the angle of view. Never really paid attention to it before," commented 3D Artist and Animator Lee, also known as leedoppo. (machine-translated)
Cameras are very important, and there's a similar tool called CamKeys for Blender by BeyondDev that allows you to control the FOV for individual objects: