[dinartem]

Good times. I was the developer at Microsoft who designed the Xbox 360 hardware security, wrote all the boot loaders, and the hypervisor code.

Note to self: you should have added random delays before and after making the POST code visible on the external pins.

[dhx]

Have you seen Tony Chen's (development lead for Xbox One security) description of the Xbox 360 reset glitch hack at [1] and the effect this (and other console exploits of that time) had on Xbox One development?

This is one of my go-to case study videos for the development effort required to architect a computer to resist attackers who have physical access.

[1] https://youtu.be/U7VwtOrwceo?t=536

[spencerflem]

Congratulations, haven't had a reason to mess with it myself, but I've heard it described online as the most secure piece of consumer hardware before or since

[jsheard]

I think you might be mixing up the Xbox 360 with the Xbox One, the former was ultimately compromised in several ways, but the latter's security has held up extremely well for 11 years and counting. The Xbox One and its successor are easily the most secure consoles ever made.

Obligatory: https://www.youtube.com/watch?v=U7VwtOrwceo

[ChocolateGod]

> The Xbox One and its successor are easily the most secure consoles ever made.

Microsoft also allowed any console to switch to developer mode and run homebrew, massively reducing the need for people to try find exploits.

[spencerflem]

Found it! I misremembered, it was making a slightly different claim

"The Xbox 360 hypervisor is probably the most secure piece of code Microsoft has ever written." from the excellent article Tony Hawk's Pro Strcpy

https://icode4.coffee/?p=954

[landr0id]

The Xbox 360 was overall a very, very secure device. While we don't know exactly how the folks who discovered the hypervisor syscall handler bug were able to get plaintext, it's theorized that it came from development kit and SDK leaks. With an SDK and dev kit someone could dump boot loaders and the HV.

Otherwise on a retail console you can't do much. The hard drives are not encrypted but all content that can possibly contain code / save data is signed. Save data cannot contain code but introduces scripting engine / save parsing attack surface, but you can't modify it without first dumping keys from a retail console.

To dump keys from a retail console you have to get code exec in the hypervisor. To attack the hypervisor you have be able to dump the hypervisor to audit it.

To dump the hypervisor you have to be able to read its contents or dump it from flash. The flash is encrypted with a per-console key (and I don't think you can sniff the bus?) and RAM is encrypted.

Realistically if it weren't for the original syscall handler bug and dev kits getting into researcher's hands, the Xbox 360 may have never been hacked.

[markus_zhang]

Stupid question, is the reason that people cannot simply dump the ROM as they do with say routers is that the rom is encrypted? But if they have the SDK they can decrypt it?

[landr0id]

The flash chip is encrypted with the console's CPU key, and the CPU key is unique per-console and encoded in efuses. So even if one person manages to dump keys they're mostly useless for hacking other consoles. The exception to this is the "keyvault" which is the console's own private key used for signing save games. You can take save games from console A and load them on console B, so console B is able to verify console A's signature based off the public key certificate embedded in the save. Microsoft had a revocation process for revoking keyvaults if they ever leaked but they just gave up once too many were in the wild.

Dev kits are keyed differently and most of the console's keys for signing / encryption are in various SDK DLLs that if you reverse engineer you can find.

[SteveNuts]

I'm curious how it fares against a modern iPhone or similar, has that ever been compared?

[saagarjha]

iPhone has a fairly different threat model (and is more valuable to attack).

[liamwire]

I have a hard time believing the ‘since’ part of that description. Intuition suggests the latest iPhone would take that crown each year.

[walterbell]

> the latest iPhone would take that crown each year

Apple continuously patches zero-day kernel exploits against the latest iOS and hardware, https://support.apple.com/en-us/100100

[StrauXX]

Sure, but there are many more people looking at iPhone security than at Xboxes. The incentives, both monetary and otherwise, are much greater with iPhones than they ever were with consoles.

[anyfoo]

That is far from the only thing that would be relevant for such a metric. (For one thing, you also have to ask which kernel.)

[walterbell]

iPhone and iPad devices have been breached by zero-days for years, unlike Xbox One.

[anyfoo]

Still simplistic. "Breached" can mean a number of things, and the incentives here are very different.

[spencerflem]

Misremembered, it was saying it was the most secure code Microsoft has written, not anyone

From the excellent: https://icode4.coffee/?p=954

[notavalleyman]

What are the reasons for why Microsoft wanted to lock down consoles to only run signed code? As a games console manufacturer, what are the business reasons for doing so? Thanks

[nemothekid]

Limiting piracy is the ongoing reason, but there is also the historical reason of the Video game crash of 1983 which led to Nintendo's Seal of Quality.

Essentially as the platform owner, you want to ensure games sold for the platform "just work", and if you have a bunch of third parties running bad software, consumers would lose faith in the platform altogether.

[mrandish]

> ensure games sold for the platform "just work"

To add a little more color to this, it wasn't solely to ensure games worked. The lesson of the video game crash was that third party publishers would make knock-off games similar to popular titles and flood the market with them at much lower cost - sometimes as low as $5 vs for a $40 for a top title. These games were generally low budget and rushed to market to capitalize on looking like a top-selling title - while being just different enough to (hopefully) avoid trademark infringement.

These games usually "worked" (as in booting up and playing), the issue was more that that they were just bad versions of the title they were ripping off due to having little development time and minimal play testing along with poorer artwork and fewer levels (thus saving ROM memory). The flood of cheap, bad versions of more popular games is credited as the main factor that killed the Atari VCS.

Another big factor was that later console manufacturers charged game publishers a license fee for the proprietary library code required for a console to run a game. This fee could allow manufacturers to sell game consoles at cost or even below cost and recoup the lost profit over time in the per game license fee.

This wasn't always the case in the early days of hardware cartridge systems. Initially, some early console manufacturers didn't charge much more than a game publisher could buy blank cartridges for from a third party. Some other manufacturers chose to generate revenue simply by building more margin into the wholesale price they charged game publishers for blank cartridges. Of course, when console manufacturers started increasing their cartridge profit margin, game publishers were motivated to use third party cartridges - which led to console makers deploying "genuine hardware" checks or, later, disc checks and encryption. Nintendo popularized enforcing their business model both technically and legally (by requiring an IP license). Today, console manufacturer business models rely on 1) Collecting per game license fees, 2) Blocking piracy, 3) Limiting game supply.

There is a lot of interesting history around how game console business models and the legal landscape evolved over time. (https://en.wikipedia.org/wiki/Atari_Games_Corp._v._Nintendo_....)

[maximilianburke]

I think it's also worth pointing out that the console makers (and developers) pour a lot more resources into ensuring that the products released for their platform are of a suitable quality than, say, phone app store gatekeepers.

A big draw as well is that people can't (within the economic viability timeframe of the games/console) hack the games on a console, meaning you get a much more predictable online experience than you might on PC.

[droopyEyelids]

There was a period of time when this was true but at least with nintendo, the eshop is full of shovelware now

[NotPractical]

Microsoft actually reversed course on this. You can make a one-time purchase to access "developer mode" and then run whatever you want. It's been suggested that this is the reason there's been less interest in hacking the Xbox. Ironically it also means you have more computational freedom on the Xbox than on the iPhone/iPad.

[girvo]

> then run whatever you want

Not quite. You (were? I don't know if this is changed) limited in how much of the hardware you could access: it wasn't 100% access. Enough for most homebrew, emulators and so on, but it wasn't carte blanche "replacement for a dev-kit" access.

[Narishma]

I don't think it much different from the official games, which also don't have full access to the console.

[klausa]

The "not-behind-the-paywall-and-NDA" GDK version is severely more limited than the invite-only GDKX.

IIRC the homebrew you can run is mostly UWP stuff? But if you want to launch a _game_, built for an Xbox, it requires to be in the program.

[girvo]

No it was quite different (again, unless they changed it? They might've, I'm long out of this scene). UWP apps had a lot more restrictions on them, the SDK was different, and you didn't have the full consoles hardware to use, compared to the "behind the NDA" SDK: it'd run slower, basically.

Still great, and good enough for most use cases

[akira2501]

> and if you have a bunch of third parties running bad software, consumers would lose faith in the platform altogether.

Famously the reason no one ever used Microsoft Windows.

[mschuster91]

> Famously the reason no one ever used Microsoft Windows.

Microsoft in its early days invested a shit ton of money and effort into backwards compatibility testing and fix development. Up until Windows 7 you could be reasonably sure that any piece of software from the Windows 95 32-bit days would still work without major issues - even 16-bit software would run under a 32-bit W7 host, only W7 x64 finally dropped support for that.

[markus_zhang]

I think they even run windows 3.1 games back then. It is very impressive. Actually I think some still do. But the ones using WinG do not, like Fury3.

[hsbauauvhabzb]

Yet Microsoft bought Bethesda

[treyd]

They sell the consoles at a loss, so if you could port your own games to the consoles instead of buying the games that they could take a royalty from then they lose money. It doesn't have to be an effective circumvention to trigger the DMCA making it illegal.

[throwaway48476]

They sell the console at a loss but make 30% on every game sold. The business model is variant of "dumping" but antitrust isn't enforced anymore.

[doctorpangloss]

If it were possible to go the wrong way on the expanding brain meme, this would be it.

Nobody is buying consoles for the hardware. 99% of the product is software. The accounting that you are using, a popular one, is so opppositely-informative that people who make consoles - and smartphones for that matter - clearly do not make decisions with it. It is 200% wrong to characterize it as dumping. Nothing is being dumped.

Here’s a simple idea for you: show me the vibrant market for Xbox 360 2005 era computer hardware. There isn’t any right? What about Xbox One 2013 era? And yet we still play games that were developed earlier than 2013, like League of Legends. The product is software. Nobody loses anything by being unable to run Linux on 2013 hardware today, and nobody loses anything by being unable to run Linux on the Xbox One in 2013 because, if they wanted cheap computers then, they had plenty to choose from!

[throwaway48476]

Both the hardware and software are products. No one selling open platform PCs can compete with negative margin console sales, which is anticompetive. If antitrust prevented dumping consoles would be more expensive and consumers would buy PCs instead. In both software and hardware the open PC platform is far more competitive which drives value for consumers.

[doctorpangloss]

> No one selling open platform PCs can compete with negative margin console sales

This makes zero sense. Both have existed simultaneously forever, and hundreds of millions of people eagerly buy both for the same households. I cannot understand your point of view here, other than invoking a word "dumping" and "anticompetive" that you are using 200% wrong. Consoles and open platform PCs do not compete with each other.

> In both software and hardware the open PC platform is far more competitive which drives value for consumers.

The market for high budget single player games exists solely because of DRM protected consoles. So this category of product, that people eagerly have paid for for decades, to the tune of hundreds of billions of dollars, would cease to exist if you required console makers to allow people to bypass DRM. Ask 20 people in the game industry and 19 would agree. I understand the core and spirit of what you are saying, but it is reflecting your aspirations for a world that doesn't exist. Markets aren't art exhibits!

[throwaway48476]

That both exist does not make negative margin sales not an anticompetive practice. Dumping is the correct term for such anticompetive behavior. Consoles are just x86 PCs. They run the same application software.

The Witcher 4 is a high budget single player game and sold 30 million copies despite having no DRM. I don't think you're familiar with the games industry.

[markus_zhang]

Just curious do we know how much loss for one hardware piece? What's the margin on extra controllers and peripherals?

[throwaway48476]

There are firms that specialize in competitive analysis and cost estimates. The margin on controllers and peripherals is fairly high.

https://www.gamespot.com/articles/ps3-manufacturing-costs-do...

[markus_zhang]

Thank you!

[brokenmachine]

They are rent seekers.

[hsbauauvhabzb]

I’m not sure that’s a fair perspective, they built a proprietary product that’s intended to be a loss leader, the opposite of that is a desktop which costs substantially more.

[brokenmachine]

It would only cost more upfront, definitely not over the life of the console.

Rent seekers don't do anything for your benefit. The definition of false economy.

The only difference between their "Proprietary product" and a PC is DRM.

[ashleyn]

business reasons, the console is sold as a loss leader and the real money is made in licensing the games. also serves as brand protection i.e. preventing poor quality third party games from tarnishing the reputation of the console.

[Lammy]

A games console provided a platform where they could more effectively argue that “their” works “““needed””” to be protected so they could farm us (people who want to run their own code on hardware they purchased) for digital-jail technologies which would never otherwise have reason to exist. Then those technologies can metastasize fully-formed over to general-purpose computing in a way that's harder to argue against. They learned with Clipper and Palladium that trying to develop jail tech on PC would be vehemently opposed.

[lyu07282]

The opposition was pointless though, like everything has TPM/IME/etc. nowadays so we lost that war awhile ago. I don't see how consoles helped them win that war though.

[RGamma]

When everything is locked down for good I'll go back to physical books. Vaporware games and media are not worth preserving anyway. And the games and media I do have free from prison can entertain me for a very long time.

Another possible (even worse) future could be cloudification of everything. Enjoy your thin client.

Of course software is bigger than entertainment which might represent a problem. We're increasingly societally locked into this digital shit.

[liamwire]

Can you speak to some of the harder or more interesting challenges you faced during that time?

[dinartem]

One challenge was that while I started working on the Xbox 360 about three years before it would ship, we knew that the custom CPU would not be available until early 2005 (first chips arrived in early February). And there was only supposed to be one hardware spin before final release.

So I had no real hardware to test any of the software I was writing, and no other chips (like the Apple G5 we used as alpha kits) had the custom security hardware or boot sequence like the custom chip would have. But I still needed to provide the first stage boot loader which is stored in ROM inside the CPU weeks before first manufacture.

I ended up writing a simulator of the CPU (instruction level), to make progress on writing the boot code. Obviously my boot code and hypervisor would run perfectly on my simulator since I wrote both!

But IBM had also had a hardware accelerated cycle-accurate simulator that I got to use. I was required to boot the entire Xbox 360 kernel in their simulator before I could release the boot ROM. What takes a few seconds on hardware to boot took over 3 hours in simulation. The POST codes would be displayed every so often to let me know that progress was still being made.

The first CPU arrived on a Friday, by Saturday the electrical engineers flew to Austin to help get the chip on the motherboard and make sure FSB and other busses were all working. I arrived on Monday evening with a laptop containing the source code to the kernel, on Tuesday I compiled and flashed various versions, working through the typical bring-up issues. By Wednesday afternoon the kernel was running Quake, including sound output and controller input.

Three years of preparation to make my contribution to hardware bring-up as short as possible, since I would bottleneck everyone else in the development team until the CPU booted the kernel.

[Zeetah]

You have an awesome memory, Dinarte!

Eric Mejdric from IBM called on Friday and said we have the chips, when are you guys getting here?

I took a red eye that night and got to Austin on Saturday morning.

We brought up the board, the IBM debugger, and then got stuck.

I remember calling you on Sunday morning. You had just got a big screen TV for the Super bowl and had people over and in-between hosting them you dropped us new bits to make progress.

I think Tracy came on Sunday or Monday and with you got the Kernel booted.

Good times!

This is Harjit by the way.

Edit: added super bowl.

[saturn8601]

OMG Harjit! I saw you in the documentary! You and the entire team are total rockstars! I just cannot fathom ever being in a position to design something that provided so much joy and happy core memories to countless people around the world...you guys did it!

Just the thought of how many people you touched with your work....just amazing! :)

Had a question if you don't mind: Can you talk about the thought process behind the power supply design? Its very large even in the super slim models. Were you following a specific design driven by the hardware architecture or were there other reasons? I always wondered about that.

[maroonblazer]

>I saw you in the documentary!

I presume you're referring to this one: https://www.xbox.com/en-US/power-on#watch

[saturn8601]

Yes! It is worth watching every second. What an amazing production.

[dinartem]

Actually Tracy never made it to Austin. He was going to fly in later in the week to continue bring-up, but since we were done by Wednesday, we just sent the chips to Redmond and he continued there. He was of course always available on the phone to answer my kernel questions I had.

[markus_zhang]

This is really some blast from the past. Can you please shed more light on the simulator? Is it interpretation or JIT? But then I realize XBOX uses Pentium III, so maybe virtualization? Edit Sorry it was XBOX 360 so it's not Pentium.

As someone who recently got interested in emulation and wrote two lc-3 emulators, would really love to learn from the masters.

[brokenmachine]

Kick ass. This kind of post is why HN is the best.

[fragmede]

damn, that's bad ass. did that simulator run on a Windows system or was it something more esoteric?

[dinartem]

I called the simulator Sbox and it was just a simple console app. I didn't implement the GPU, so no graphics just the hypervisor and kernel and some simple non-graphics apps. I made it so that you could build the Xbox 360 kernel on your windows machine, then just run sbox.exe and it would automatically find the just built kernel image targeting the PPC64 and boot it. Then if you typed control-C it would drop into the kernel debugger as a sub process, and you could poke around at the machine state as if it were the real Xbox hardware, showing all the PPC instructions and registers. It was a lot of fun writing it, and quite useful.

[Zeetah]

You should also talk about the lwarx/stecx bug. IIRC - in the first version of the chip there was a bug in one or both of these instructions. Your code booted on SBox but didn't on the hardware. You compared the two and then figured out it was these instructions.

You filed a bug report and then dug into them and used SBox to figure out what must have been going wrong.

The chip supplier came back with a workaround and within five minutes you simulated it on SBox and said it wouldn't work, why, and then said how it should be fixed.

The supplier didn't believe you as yet. And you worked out a workaround so we could be unblocked. Two weeks later they agreed with your fix...

[maximilianburke]

I recall an issue when trying to use lwarx/stwcx on Xbox 360 directly that the compiler (or maybe even the kernel, on program load? it's been a while) raised an error and said to use the Interlocked intrinsics instead -- is that related?

[fragmede]

That sounds like a lot of fun! What was your career like before that, that led you to be in the place to do such fun things?

[kridsdale1]

This is the coolest HN post I’ve read in months.

Cheers.

[cbanek]

Small world! I worked on Yellow Door / Golden Gate automation for releasing 360 titles and patches to prod, and the beta group / KDC service code.

[saturn8601]

You sir are an inspiration! I am but a mediocre Angular developer and reading this has me in complete awe! The kind of drive you must have had to get this done well I dont know how people manage to do it but it is so cool to see! :)

[n144q]

> we knew that the custom CPU would not be available until early 2005

Sounds a little bit like the situation with Xbox Series? The SDKs were released late because Microsoft was waiting for certain features in AMD APU

[bananaboy]

Amazing! Thanks for sharing! What sort of things are you working on now?

[Dracophoenix]

How did you feel about teenagers and college students exploiting holes in your work? Were you impressed, disappointed, amused, etc.?

Oh and I'd just like to say thank you for your contribution to my childhood/adolescence.

[dvdbloc]

What was the culture like working on this project and back in those days? I’ve always been fascinated by the development of consoles, especially the story of the 360. Any sources you recommend to learn more? I thought the Microsoft documentary on Xbox was the best I’ve found so far.

[jolan]

The Winchester revision is still considered unhackable afaik. It's crazy how many Xbox 360 revisions there were compared to other consoles.

[saturn8601]

Was it ever explained why? This was an unanswered question I always wondered about from time to time. They must have done something to remove RGH capability?

[jolan]

Yes that revision is patched to specifically counter RGH. Microsoft disabled the ability to get the precise timing needed from the CPU and also added more filtering/robustness so the system will reset properly instead of getting into the inconsistent state of the old revisions.

[landr0id]

Wait until you read Grimdoomer’s (yet to be published) write-up on his recent hypervisor exploit that works on the latest OS: https://x.com/grimdoomer/status/1847497275289063676

It’s going to be a doozy

[vlovich123]

I feel like random delays would make the glitch attack harder but it would still be possible given enough attempts. Seems like the bigger issue is that you can glitch the CPU reset line which corrupts the processing rather than having no effect or resetting the CPU.

[kaoD]

I assume those are probably very hard to fix since (again, I assume, I'm just a hobbyist in the hardware space) that sort of glitch relies on propagation delays (e.g. a short burst triggering some latches but not others, or triggering the latches in some specific synchrony).

Can anyone confirm if I'm on the right track with my guess?

[LPisGood]

I’m surprised this was a task that only took a single developer!

[cbanek]

Having a single developer allows fewer offices with their windows completely covered with newspaper. Plus, there's one person doing everything, which can be a lot better than two with people who have different ideas of how to make the system work together.