[nkurz]

You are right. The largest Xeon Phi's have 16GB of GDDR5 memory on the card, and the 31S1P on sale has 8GB. The quoted speeds are for 16-channel transfer between the card's on board memory and the cores on the same card. Transfer from main memory back and forth to the card is over PCIe v2 x16 which has a theoretical max of 8 GB/s --- about 3% of the maximum transfer rate on data within the card.

So if your working data set fits on the card (less than 8/16 GB), or can be partitioned so that it fits on multiple cards, you can potentially get great performance from Phi. But if it's larger than this, especially if it is less than 1TB and would thus fit in the host's RAM, transfer from the host to the card will likely destroy any performance advantage.