Once partial reconfiguration works and the FPGA can access main memory directly I see a lot of use cases. Imagine applications reconfiguring the FPGA in the blink of an eye to optimize their own algorithms.
Why? To make an FPGA do what you want you need to be able to reconfigure it. If you have reconfiguration capability you need to have remote code execution. And in that case you have already lost.
As in, the FPGA would have to be carefully segmented so the accelerator couldn't be used to access memory it shouldn't have access to.
I don't think it would happen in a general purpose chip but I could see it happening in a smaller one like the exploits christopher Domas demonstrated against some embedded X86 cores.
Why though? Your Integrated Intel or AMD GPU can also access all of your memory. I don't see how an FPGA provides any additional attack vector. As I said you'd need code execution privileges anyway and once you have that your system is already owned.
Yes through the PCI bus not directly. You don't want to have that latency. You want a unified model. Like Intel GPUs that can access main memory, or the FPGA being another endpoint in AMDs infinite fabric architecture. That exists as well in SoCFPGA boards. But not in the mid or high performance segments.
Back when AMD released the first Opteron CPUs there was a vendor selling an FPGA that would plug into an Opteron socket along with the IP to implement HyperTransport in the FPGA.