Well, the top of the range FPGAs are priced two orders of magnitude above the top of the range GPUs, so in terms of Tflop/$ GPUs will win in many cases.
Given that most projects, the power costs are much higher than the upfront equipment costs and that the dominating factor on computational density and interconnect speeds is our thermal budget, I'd think it's really a question of Tflop/watt.
The cost of each watt of grid power for 2 years is about $1. A high-end GPU costs $3000 and burns 200W, so power costs $200 over 2 years, or 6% of the total cost. I can't think of any high-performance computing semiconductors costing less than $1 / watt.
What systems can you point to where the power cost over the time-to-obsolescence exceeds capital cost? Besides Bitcoin mining.
You're correct, I had receive wrong information and never really thought it through in regards to high end systems.
Just about the only systems where it makes sense to talk about the power budget being relevant are where you're going in on base commodity systems and talking about a 3-4 year cycle time. (And maybe weird cases where we're having to meet power budgets of existing deployments.)
I was comparing situations that had already made a FLOPS/dollar decision because of constraints on other resources (cheap hardware, lots of it, TONS of storage, high sync latency), and so I guess both falls outside traditional HPC and is a secondary concern.
In terms of power, there is a largely unexplored but yet very interesting world of the mobile GPUs. Project "Mont Blanc" is about to dig into this area: http://www.montblanc-project.eu/
But, yes, I'm very enthusiastic about the Altera acquisition by Intel, it may drive prices down and we'll probably see FPGA-enhanced Xeons soon.