[kenny-log_ins]

The technical challenges posed by the SKA really are astoundingly huge. The data generated by these dishes will require much higher data throuput than even the fastest optical links between CERN and its tier 1 sites (40 Gb/s), and they will be in the middle of a remote desert. Then there is the scale of compute and storage needed to analyse the data, which would be totally unfeasible with the tech we have today. It's an exciting time to be working in scientific computing!!

[CHY872]

Well, that's as much because CERN doesn't need additional throughput than anything else; technically it's a solved problem, which makes it one of the easier ones to do something about :)

We can quite easily send 40Gb/s over a single fibre (I feel like I've seen over a terabit done...), and the undersea cables (more inhospitable than the remote desert) are far, far higher bandwidth.

Consuming the data in a timely manner may be tricky, though!

[elithrar]

> and the undersea cables (more inhospitable than the remote desert) are far, far higher bandwidth

Terrestrial cables are likely harder to lay: you have to worry about land access—national parks, native title, private land—and the W.A. outback is an extremely challenging environment to work in.

Getting dedicated DWDM fibre to Murchinson is a big project unto itself.

[kenny-log_ins]

The optical private network from CERN to the tier 1 site where I work was upgraded from 10 Gb to 40 Gb between the first and second runs of the LHC. depending on what goes on during the second run that might need to be increased again. Building high bandwidth infrastructure for use by lots of (paying!) customers is probably more viable for large corporations than it is for science, where you have a budget at the start rather than having things pay for themselves over time

The storage will need to constantly increase in that time, too.

[CHY872]

Yes, of course, but this seems to be a problem where the solution is easily costed, is commercially available etc, whereas I'd imagine that the rest of the project is filled with a whole load of world-firsts.

[toomuchtodo]

Tier 1 = fermilab?

[seanp2k2]

The NSA can do it, maybe they should contribute some engineering effort to help scientific causes like this and gain some good PR in the process.

[Osmium]

CERN had a similar problem too, and the only solution was just to throw away a lot of their data. Do you expect this might happen with SKA, or is the hope that they can capture everything for later processing?

[kenny-log_ins]

It depends - the amounts of data involved are mammoth by today's standards, but the idea is that we will develop new methods and tools to make the project feasible. The announcement and funding of the SKA will probably help this process along too, as specific challenges are addressed throughout the programme

[Xophmeister]

From the article:

    There will be far too much data for it all to be simply recorded.
    The plan is to "use an algorithm to reduce that data down to a
    smaller magnitude that you can then process further on a super
    computer", Morgan says.