Those things actually exists and are called NPU (Neural Processing Unit). Many recent ARM CPU have them (Apple, Qualcomm, etc.) and even new RISC-V CPU. They're different from GPU in being even more constrained in programming requiring usage of a fixed function vendor-supplied library.