Is this due to the game being poorly optimized for the PS4, or because the TV adds some lag? I have heard that TV lag can be reduced by turning on "Game mode" or equivalents.
No, other games perform fine. TV correctly set up (I know what you mean, some TVs have frame interpolation to fake 60 fps which adds lag) and gamepad connected via cable.