You have to take into account that Titan are super cheap compared to FPGA. Big FPGA boards for HPC can easily cost between 5k$ to 10k$. If you compare to GPU in the same price range, you end-up with K40 or K80, who have a peak at 4.3TFLOPS SP and 5.6TFLOPS SP respectively, much higher than Stratix 10. Moreover, FPGA are not really good at double precision FP, which is important in many HPC area.
At the end of the day, the important metric is FLOPS/$, and more importantly what you can achieve for your application and tooling and ecosystem. Many scientists are not computer science experts, and many HPC codes are legacy simulations which can be hard to port and re-validate.
In my experience, FPGA are still a nich accelerator vs GPU. And I am not even talking about future Xeon Phi generations.
And of course, when talking about HPC you should not forget the elephant in the room: standard Xeon...
My work intends to change the aspect of having to be a computer scientist in order to leverage the power of an FPGA by using Haskell/CLaSH as a HDL which is close to mathematics.
Furthermore, the verification of the designs is simplified a lot by checking directly in Haskell over generating VHDL testbenches and then running an additional simulator tool.
Lastly, I hope that with the recent acquisition of Altera by Intel, some of the other issues you mentioned (mainly floating point performance) additionally with some tooling issues will be addressed as well.
I understand that, and having a DSL is definitely a good idea, but you need to create a community, which can be hard (and NVIDIA seems to be good at it). I didn't want to be deceptive, I just wanted to highlight that it was not just a matter of peak FLOPS (in fact it never is - as an engineer working on another niche accelerators I know it too much ;) ).
Well, the top of the range FPGAs are priced two orders of magnitude above the top of the range GPUs, so in terms of Tflop/$ GPUs will win in many cases.
Given that most projects, the power costs are much higher than the upfront equipment costs and that the dominating factor on computational density and interconnect speeds is our thermal budget, I'd think it's really a question of Tflop/watt.
The cost of each watt of grid power for 2 years is about $1. A high-end GPU costs $3000 and burns 200W, so power costs $200 over 2 years, or 6% of the total cost. I can't think of any high-performance computing semiconductors costing less than $1 / watt.
What systems can you point to where the power cost over the time-to-obsolescence exceeds capital cost? Besides Bitcoin mining.
You're correct, I had receive wrong information and never really thought it through in regards to high end systems.
Just about the only systems where it makes sense to talk about the power budget being relevant are where you're going in on base commodity systems and talking about a 3-4 year cycle time. (And maybe weird cases where we're having to meet power budgets of existing deployments.)
I was comparing situations that had already made a FLOPS/dollar decision because of constraints on other resources (cheap hardware, lots of it, TONS of storage, high sync latency), and so I guess both falls outside traditional HPC and is a secondary concern.
In terms of power, there is a largely unexplored but yet very interesting world of the mobile GPUs. Project "Mont Blanc" is about to dig into this area: http://www.montblanc-project.eu/
But, yes, I'm very enthusiastic about the Altera acquisition by Intel, it may drive prices down and we'll probably see FPGA-enhanced Xeons soon.