There are already some companies doing specialised inference hardware, Cerebras Systems for example. Such designs are still early days and I wouldn't be surprised to see more innovation there. Though because custom silicon design takes time I expect a multi-year cycle.
For training, not sure. But even if training runs on GPUs, once you have the model the main cost is inference.
For training, not sure. But even if training runs on GPUs, once you have the model the main cost is inference.