[lsc]

Personally, I'd rather have open-source firmware for spinning disks. There's a lot of room for optimization of spinning disks. If the firmware could pass through more realistic cylindars, and if the firmware would quit silently re-mapping bad sectors, there's a lot of performance we could get out of spinning disk.

I mean, on the 'remapping bad sectors' bit, right now? to the OS, it looks like you are doing a sequential read, but if there was a remapped sector in there? you're doing a random read. We'd be much better off, at least on server systems with decent raid subsystems, just handling the bad sectors in software.

Right now, I pay almost double for my spinning disk because I want slightly better firmware that is designed to fail outright rather than retry, because all my disks are in raid. some consumer disks allow you to adjust the time-limited error recovery paramiters, but in my experience, it's super unreliable.

[deelowe]

Is there any data on this? As someone who works closely with hard drive test teams, I've not seen any indication that there's much room for improvement on spinning disks (except incremental gains from increased platter density). The proprietary firmwares already do a lot to optimize performance and each vendor has their own secret sauce, esp for enterprise solutions.

[lsc]

The key to spinning-disk performance is to access it in a sequential manner. The effect of firmware that does things like transparent sector remapping is that you can never know what, exactly, is sequential and what is not.

The problems I have with stock firmware isn't that the stock firmware isn't good... it's that the stock firmware hides a lot of information that the rest of my system could use to make the access to the disk more sequential. It also makes the assumption that I want this drive to function by itself without complex zfs-style software on top of it that can handle errors.

The idea being that the drives are setup with a firmware that makes the drives okay in a configuration where it's a single drive running ms-dos. In a multi-drive, redundant configuration with something like zfs doing the raid, a lot of the "features" there (retrying for a long time on errors, remapping bad sectors) are problems that are handled far better at a higher level.

So... if your point is that I could just spend the money and get a storage appliance from emc or netapp, you are right, but those things cost on the order of 10x what local storage costs.

If your point is just that I'm being cheap, that's fair too, and that's probably why drive makers don't want open-source firmware; with closed source firmware, they can charge extra for the feature-set that is optimal for a RAID configuration, and less for the feature-set that is optimal for a single-disk configuration. Even though both of those feature sets might be pretty good for what they do, they are largely the same drive, physically, and even if they weren't, I'd be happy to have a somewhat less-reliable drive that reliably returned errors rather than hanging, if it saved me the rather large difference in price between 'consumer' and 'enterprise' drives. - right now, I'm paying the extra for the 'enterprise' drives even thought the overall reliability of consumer drives would be just fine for my use.

[deelowe]

All I'm saying is that we work closely with drive vendors and test tons of drives at work. Most of what you're claiming here seems like speculation. For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered. Also, drive vendors spend tons of R&D on bin packing both on the physical disk and the data stream to try and eek out the most performance possible. Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.

Open firmware would be great for various other reasons like being able to have confidence in a secure erase solution or being able to have better telemetry to predict failures. That said, I wouldn't put some huge secret performance gains at the top of that list.

[lsc]

> For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered.

the remapping I'm talking about happens when a sector goes bad. if sector 5 fails, it's remapped to the bit of disk set aside as spares, no? no secret there. but when my computer reads sector 1 through sector 10, my computer thinks it's reading sequentially... but really there's a big random seek right in the middle of it. By my understanding, that happens every time I go to read sector 5.

>Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.

The point here is that it's not simple, and it's not a conspiracy or anything... and a lot of this wouldn't be a win for computers running windows with a single disk, and the big customers do have access to this sort of thing; like I said, emc and netapp and the other big vendors have their own custom firmware.

My guess is that the vendors don't want to do anything that requires the user to run special software, and all the changes I'm talking about would end very badly for the user if they weren't running a filesystem/raid system that was built to handle that sort of thing.

[deelowe]

We have the ability to run special software and they don't give us these features. I'm sure they would if they thought would help.

On the remapping of sectors, that happens very infrequently. Look up smart reallocation count indicators for more info on this. Once a few percentage points of space has been reallocation, smart will assume the drive is failing.

[SamReidHughes]

You don't really need open source firmware to figure out that sector 5 is remapped. You just need a clock.