[shasheene]

It seems pretty clear that PCIe-based solid-state drives using the NVMe protocol (and U.2 or M.2 connectors) will have much higher performance than AHCI/SATA over the next 5+ years (for both latency and throughput).

Since coreboot (with TianoCore or SeaBIOS) and the impressive nouveau and radeon projects, there's fewer places binary blobs can hide on a modern computing system.

One of the bigger exceptions is the drive controllers (which processes AHCI/NVMe messages and things like wear levelling). Besides restrict freedom and performance experimentation, it's also a security issue, given the scope for man-in-the-middle attacks [1]

Now at the start of NVMe's reign, it's probably an especially good time to start a project to make an open-source replacement to the proprietary firmware blobs of SSDs/flash memory! :)

[1] http://www.bunniestudios.com/blog/?p=3554

[lsc]

Personally, I'd rather have open-source firmware for spinning disks. There's a lot of room for optimization of spinning disks. If the firmware could pass through more realistic cylindars, and if the firmware would quit silently re-mapping bad sectors, there's a lot of performance we could get out of spinning disk.

I mean, on the 'remapping bad sectors' bit, right now? to the OS, it looks like you are doing a sequential read, but if there was a remapped sector in there? you're doing a random read. We'd be much better off, at least on server systems with decent raid subsystems, just handling the bad sectors in software.

Right now, I pay almost double for my spinning disk because I want slightly better firmware that is designed to fail outright rather than retry, because all my disks are in raid. some consumer disks allow you to adjust the time-limited error recovery paramiters, but in my experience, it's super unreliable.

[deelowe]

Is there any data on this? As someone who works closely with hard drive test teams, I've not seen any indication that there's much room for improvement on spinning disks (except incremental gains from increased platter density). The proprietary firmwares already do a lot to optimize performance and each vendor has their own secret sauce, esp for enterprise solutions.

[lsc]

The key to spinning-disk performance is to access it in a sequential manner. The effect of firmware that does things like transparent sector remapping is that you can never know what, exactly, is sequential and what is not.

The problems I have with stock firmware isn't that the stock firmware isn't good... it's that the stock firmware hides a lot of information that the rest of my system could use to make the access to the disk more sequential. It also makes the assumption that I want this drive to function by itself without complex zfs-style software on top of it that can handle errors.

The idea being that the drives are setup with a firmware that makes the drives okay in a configuration where it's a single drive running ms-dos. In a multi-drive, redundant configuration with something like zfs doing the raid, a lot of the "features" there (retrying for a long time on errors, remapping bad sectors) are problems that are handled far better at a higher level.

So... if your point is that I could just spend the money and get a storage appliance from emc or netapp, you are right, but those things cost on the order of 10x what local storage costs.

If your point is just that I'm being cheap, that's fair too, and that's probably why drive makers don't want open-source firmware; with closed source firmware, they can charge extra for the feature-set that is optimal for a RAID configuration, and less for the feature-set that is optimal for a single-disk configuration. Even though both of those feature sets might be pretty good for what they do, they are largely the same drive, physically, and even if they weren't, I'd be happy to have a somewhat less-reliable drive that reliably returned errors rather than hanging, if it saved me the rather large difference in price between 'consumer' and 'enterprise' drives. - right now, I'm paying the extra for the 'enterprise' drives even thought the overall reliability of consumer drives would be just fine for my use.

[deelowe]

All I'm saying is that we work closely with drive vendors and test tons of drives at work. Most of what you're claiming here seems like speculation. For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered. Also, drive vendors spend tons of R&D on bin packing both on the physical disk and the data stream to try and eek out the most performance possible. Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.

Open firmware would be great for various other reasons like being able to have confidence in a secure erase solution or being able to have better telemetry to predict failures. That said, I wouldn't put some huge secret performance gains at the top of that list.

[lsc]

> For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered.

the remapping I'm talking about happens when a sector goes bad. if sector 5 fails, it's remapped to the bit of disk set aside as spares, no? no secret there. but when my computer reads sector 1 through sector 10, my computer thinks it's reading sequentially... but really there's a big random seek right in the middle of it. By my understanding, that happens every time I go to read sector 5.

>Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.

The point here is that it's not simple, and it's not a conspiracy or anything... and a lot of this wouldn't be a win for computers running windows with a single disk, and the big customers do have access to this sort of thing; like I said, emc and netapp and the other big vendors have their own custom firmware.

My guess is that the vendors don't want to do anything that requires the user to run special software, and all the changes I'm talking about would end very badly for the user if they weren't running a filesystem/raid system that was built to handle that sort of thing.

[deelowe]

We have the ability to run special software and they don't give us these features. I'm sure they would if they thought would help.

On the remapping of sectors, that happens very infrequently. Look up smart reallocation count indicators for more info on this. Once a few percentage points of space has been reallocation, smart will assume the drive is failing.

[SamReidHughes]

You don't really need open source firmware to figure out that sector 5 is remapped. You just need a clock.

[pjc50]

How do you propose to do such an open-source firmware? If you do it by reverse engineering targeting existing hardware, you're continually vulnerable to obsolescence and (spurious but inconvenient) legal threats. If you do it by building new hardware, you discover just how small the market is for more expensive small-batch open hardware is.

[walterbell]

Don't forget Intel SGX, which arrives with Skylake over the next few months, https://www.reddit.com/r/hardware/comments/3nn33x/intel_to_b...?

[JoshTriplett]

http://www.openssd-project.org/wiki/The_OpenSSD_Project

[Keyframe]

Considering next 5 years, it will be interesting to see what will happen once Intel 3D Xpoint comes to market.

[ris]

Perhaps nothing. The closer you look the less you see with xpoint.

Interesting article: http://www.tomshardware.co.uk/intel-micron-3d-xpoint-updates...

[Keyframe]

Interesting article indeed. I wasn't aware of the politics and rivalry with Samsung behind it. I was kind of rooting for near-future RAM and disks becoming one. Looks like, instead, we will get Rambus v2 story.

[zokier]

My personal preference as a solution for this "problem" would be to make the drives dumb enough that the firmware wouldn't matter that much anymore. Something like ONFI flash with UBIFS. Flash chips are not such unique snowflakes that a generic software with some tweakable parameters could not manage them.

[JoshTriplett]

> Flash chips are not such unique snowflakes that a generic software with some tweakable parameters could not manage them.

Take a look at any of the recent reviews comparing drives with very similar flash hardware, yet with wildly different performance characteristics. Firmware matters.

I'd certainly like to see such devices with Open Source firmware, but don't discount the difficulty of the problem.

[nitrogen]

Also, we want to offload as much tedium as possible to secondary processors so the CPU is fully dedicated to our workloads.

[tracker1]

The same was once said for modems and audio adapters...

[pkaye]

The flash data sheets are actually just the starting point for using NAND flash. It is not just signal timings and command sets. There are lots of quirks you have to work around to make it reliable and usable which changes with each generation of NAND.

[kozukumi]

Is there such a thing as a drive controller rootkit that can intercept/process data? Something that can understand common FS such as NTFS and they "lie" about what is really there? Perhaps inject additional executable data into an exe that is called? Could it break hardware encryption on the drives?

[frozenport]

Yes, indeed. This has been done as a proof of principle [1]. I don't think its being used in mainstream malware.

[1] http://spritesmods.com/?art=hddhack&page=5

[nitrogen]

I seem to remember Snowden docs indicating that the NSA had HDD firmware hacks in their toolkit, and that such hacks are an example of what they mean by "advanced persistent threat".

[frozenport]

Indeed, additionally I recall that the HP server's Iran used had some similar security issue on the RAID(?) controllers. But I hate to write stuff on HN without a source :-)