Hacker News new | ask | show | jobs
by 33985868 1278 days ago
TL;DR yes and no (thanks to DSPs, technically some were an integral part of the GPU):

In the era of semi-programmable graphics pipelines, of which there was the N64, PS2 and GameCube, you would not be able to solve general-purpose problems using only that pipeline: given a handful of colour mixing formulas, registers, textures and vertices to control the pixels to write back to memory, I fail to come up with any tangible problem you could hack them into solving.

However, what the 3D generation of video game consoles had to bundle with the hardware in order to process complex scenes at interactive frame rates were digital signal processors (DSPs). Think of them as regular processors, but with an extra vector unit, allowing you to crunch enormous amounts of data in parallel (single instruction, multiple data instructions, or SIMD). This was essential to perform geometry transformations, which benefit greatly from parallelism, in order to build graphics commands to pass down to the rasteriser.

I am unsure if the SEGA Saturn and PS1's vector-accelerated instructions are sufficiently general-purpose to allow them perform any computation you wish to do, but at least starting with the Nintendo 64, you could write microcode to accelerate any task you'd like, provided you were brave enough to deal with the harsh memory constraints of microcode and obscure documentation.

To give concrete examples, the PS1 came with a dedicated chip (the MDEC, or motion decoder) to decode MPEG video frames from the CD directly into the VRAM to display on screen.

While the N64 doesn't have dedicated video decoding chips, its signal processor was designed in such a way to allow microcode to do an equivalent job (not to mention YUV texture decoding on the rasteriser's side).

So this did not stop a talented Resident Evil 2 developer fresh out of school to write an MPEG video decoder microcode to be able to cram two CDs worth of video data (around 1 gigabyte, down to a 64 megabyte cartridge), of course with other smart decisions like data deduplication and further compression/interlacing.

Another one was Rare developers writing an MP3 audio decoder microcode to store large amounts of dialogue.

And finally, in recent times, a developer managed to write an H.264 video decoder microcode, for a machine that was released almost a decade prior to this codec's birth.

You can also accelerate physics and anything else that would benefit from SIMD, really, and in fact, while more modern CPUs integrate SIMD instructions in their tool chest that compilers can take advantage of, the PS3's Cell processor was a brief throwback to the old hardcore ways, before GPGPU became king.

You could almost treat the DSPs as the compute units of modern GPU architectures: they definitely processed the vertices, and nothing stops you from adding a "vertex shader" pass, however, because it's not directly integrated into the graphics pipeline, it's harder to emulate a true pixel shader, you might be limited to full-screen pixel shading since there's no feedback from the rasteriser about which pixels exactly get written to memory.

3 comments

> You could almost treat the DSPs as the compute units of modern GPU architectures: they definitely processed the vertices, and nothing stops you from adding a "vertex shader" pass, however, because it's not directly integrated into the graphics pipeline, it's harder to emulate a true pixel shader, you might be limited to full-screen pixel shading since there's no feedback from the rasteriser about which pixels exactly get written to memory.

Actually, you could probably just use the depth buffer as write-only with constant values to identify objects as different kinds, and use a "pixel shader" pass to apply effects on those annotated pixels, but I assume this might be a pretty expensive technique.

... Now that I think about it, it's not unlike modern rendering engines.

Thanks for giving such a thorough explanation with examples.

I know most of this stuff anecdotally but I didn't start working with 3D until programmable shaders were already becoming commonplace.

You're very welcome.

Computer graphics has a very interesting history, and like many things, there's a lot to learn from studying it. Surprisingly, many techniques and principles are still relevant to this day.

Thankfully, the Internet is full of documentation, post-mortem accounts and reference implementations to learn more about all this.

Here's a bunch of random noteworthy things and references I forgot to link (sorry for potentially digging rabbit holes):

The best information tends to come from official documentation detailing almost everything you want to know about the inner workings of systems. Additionally, unofficial community documentation can also be of great quality and complement official sources.

The architecture overview posts from Rodrigo Copetti (https://www.copetti.org/writings/consoles/) pack a lot of accurate information at a glance, and are a great starting point if a specific topic piques your interest.

The Resident Evil 2 microcode post-mortem: https://ultra64.ca/files/other/Game-Developer-Magazine/GDM_S...

Improved lighting, reverberation, Dolby surround and MP3 microcodes from Rare: https://youtu.be/jcIupBUAy98?t=41

H.264 decoding microcode: https://www.reddit.com/r/n64/comments/gpwxx0/my_dragons_lair...

The Reality Signal Processor's programming guide: https://ultra64.ca/files/documentation/silicon-graphics/SGI_...

Emulators' progress reports can reveal a lot about the details and intricacies required for accurately replicating these systems' features: https://dolphin-emu.org/blog/ is one such amazing source of information, but all other major emulation efforts have equally interesting content.

Transparency sorting of surfaces (https://en.wikipedia.org/wiki/Order-independent_transparency) is a hard problem to solve for traditional scanline renderers (most PC/console GPUs), to the point that even today's releases can ship with somewhat obvious rendering errors.

On the other hand, tiled renderers (used in the Dreamcast, mobile hardware, and Apple silicon) are able to solve this problem due to their very nature, albeit trading another set of drawbacks, though the completely different hardware approach is a nice read (https://en.wikipedia.org/wiki/Tiled_rendering).

Someone once shared a video demonstrating a 3D package from ancient Lisp machines (https://youtu.be/gV5obrYaogU?t=30), and it was almost shocking to see how many things were familiar and done right from a modern perspective.

Reimplementing new techniques on old hardware. For example, someone implemented real-time shadows and toon shading on N64 (https://youtu.be/VqDAxcWnq3g).

For fun, you can grab RenderDoc (https://renderdoc.org/) and copies of your favourite games to analyse their frames (even via emulation): see how developers implement or fake visual effects and generally how these games render their world.

For instance, Dolphin emulates the GameCube's semi-programmable texture environment unit (TEV) via a pixel shader, and its shader source code is directly visible and editable, with the resulting output shown in the texture viewer. With the aid of textures, you can implement refraction, reflection and heat haze, among other effects.

Retro Game Mechanics Explained talks about both hardware and software concepts: https://www.youtube.com/@RGMechEx/videos

Thanks for the massive reply. Lots of cool stuff, and a few I didn't know about!

* I use RenderDoc a lot both for fun and at work.

* I've read the Dolphin blog every now and then.

* I love Rodrigo Copetti's website and remember reading everything it had about Nintendo consoles when I first found the site.

* I'm intimately familiar with OIT and used the AMD slides about OIT with linked lists as inspiration for the particle simulation I wrote as my Master's thesis.

* I'm familiar with tiled rendering and have worked with hardware which uses it

All the other stuff you shared is new to me and I will definitely be digging into it!

Glad you appreciate it.

Fascinating, as are your website's write-ups, thank you for sharing your experiences to the world, awesome work :)

The write up of the RE2 MPEG decoding was thrilling (even though I'd seen MVGs coverage before).

But the H264 decoding is just surreal. Imagine if someone whipped that out in 1996. Is this what they mean when they say sufficiently advanced technology is indistinguishable from magic? :)

Right ?! :p
Amazing answer thank you so much!