When you delete the config it resets everything to defaults. Restarting the game can't change the config. That means you are doing something to the config that is causing your problems.
Tested again and I made sure to absolutely not do anything to the config. I restart the server and the game, and I get the problem again. Changing the zoom sensitivities in the F12 menu doesn't seem to change anything either, everything just uses the same sens as 1x.
UPDATE: It seems changing the load order of mods to have the FOV fix load last fixed it. I'm not sure what caused it but that fixes it. Thanks for your time!