The Talos II I'm typing this reply on is just my regular workstation. I have a dual-4, which is 32 threads, and it runs Fedora. Most things work just fine and it's a very nice daily driver. I'll probably upgrade it to a dual-8 in the near future for even more parallel goodness.
I also have a Blackbird, which does streaming video in the home theatre and is also a test system. It's a single-4, but it can take an 8-core part if desired.
With that kind of hardware, is hardware decoding and encoding of media a possibility? A DLNA server with hardware transcoding is something I've been thinking about recently.
You can use AMD graphics cards with their open-source drivers. I'm not sure how well hardware-accelerated video works with them but it should work the same as it does on x86 Linux. Multimedia is probably one of the weaker areas for Power since there's so little focus on writing Altivec SIMD code for the platform from the larger open-source multimedia community (most of their focus is understandably on x86 and ARM), though that wouldn't impact hardware en/decoding.
They are primarily intended for people who need lots of threads for high-end workstation uses (see Phoronix's benchmarks[1]), people who want open computers (whether for security/auditability or ideological reasons), or just people who want something different from the mainstream.
I also have a Blackbird, which does streaming video in the home theatre and is also a test system. It's a single-4, but it can take an 8-core part if desired.