[Lramseyer]

> One of the major valid concerns that were raised with this configfs interface was security as it opens up the interface to users for modifying the live device tree.

This has always felt like a gaping security hole waiting to be explored.

Modern, high end FPGAs have a feature known as Raw SerDes, which in essence allows you to bypass a PCIe or Ethernet controller and use those lanes (yes, PCIe lanes) to your heart's desire ...provided you can design a working communication protocol. Difficult, but not impossible by any means.

So if you wanted to, you could design your own PCIe controller and give it whatever device ID, vendor ID, memory space, or capability space you want! Normally these things are not writable on a PCIe controller. But if you designed your own, you could write them to whatever you want and spoof device types, memory spaces, or driver bindings, and probably get yourself access to memory you shouldn't be touching. While I don't know how the linux kernel would handle these potentially out of spec conditions, it never sat right with me from a security standpoint.

[mschuster91]

> But if you designed your own, you could write them to whatever you want and spoof device types, memory spaces, or driver bindings, and probably get yourself access to memory you shouldn't be touching.

Not in a system with a properly configured IOMMU unit. That stuff got some serious attention back in the old Thunderbolt 2 era, when people discovered that yes, it's PCIe under the hood and yes, having no IOMMU protection yields an attacker an instant-0wn.

[defparam]

Sure, but aren't you connecting your general purpose serdes to a peer PCIe controller? I don't understand why having raw serdes control is a security concern in this regard unless you are trying to find exploits at the physical layer...

In any regard, a lot of threat models (including mine) consider installing hardware (especially an FPGA) as a trusted action.

[Lramseyer]

The thing is, the PCIe EP on the FPGAs uses the general purpose SerDes that are routed to the PCIe controller in the bitstream. So if you were to load a different malicious bitstream (which is admittedly a challenge in it's own regard) You could turn the FPGA into a malicious PCIe device.

[opello]

Is the concern the idea that as FPGA fabric is included in more devices, some hypervisor escape is going to present this as additional attack surface?

Otherwise if it's configfs you're root on the system and unless it's integrated peripherals you plan to attack you probably have finer grained hardware context to imply physical access... which seems to minimize the farther reaching, generalizable concerns?

[defparam]

If physical (evil maid attacks) are not in scope I fail to see the concern. To turn the FPGA into a malicious device you would have to gain root access to the system hosting it. So by the time the attacker is able to gain the ability to program the device, there is little need to even make it malicious. One could argue that it adds persistence vector to malware, except that the device likely will get reprogrammed over and over during normal operation. If malware authors wanted persistence they would likely target firmwares of random flash roms on chipsets and commodity PCIe cards that are less likely to be re-programmed. Lastly, the only other valid concern possibly more dangerous than root access is perhaps a remote attacker programming a bitstream to completely fry the FPGA faster than the power regulators can react and thus killing an expensive chip. That one is concerning.

[LargoLasskhyfv]

Shouldn't that be solvable by extending mandatory access control frameworks to the IOMMU?

[adwn]

The built-in Gigabit transceiver cores, which you'd have to use for the PCIe protocol, are connected to very specific IO pins on the FPGA. If the PCIe slots on your mainboard aren't already routed to those pins, then the FPGA will never be able to "bypass" the regular PCIe or Ethernet interfaces. Conversely, if they are connected to those pins, then the regular PCIe and Ethernet interfaces won't be able to use that PCIe slot. So no, your security concerns are unwarranted.

> a feature known as Raw SerDes

I have never heard anyone use the term "raw serdes" for hard transceiver IP cores.

[Lramseyer]

You're attaching your design to the Mac layer inside the FPGA, not to the IO pins, so it's the PIPE interface or something similar that you would need to communicate with. And yes, you can bypass the PCIe or Ethernet controller on various models of FPGAs.

[krupan]

Sorry, but it's still not clear what exact attack scenario you are envisioning. I have PC with a motherboard that has a CPU and am FPGA. I load my custom nefarious PCIe core onto the FPGA that bypasses the built in PCIe core. Now what? What is my PCIe core actually connected to?

[davrosthedalek]

To make the FPGA actually useful, it probably is connected to the PCIe lanes. Since PCIe isn't really a bus anymore, it's not clear what is possible, but I believe a PCIe device principally can access all of memory (via DMA or similar)? Maybe an IOMMU can protect that, but I would be very surprised if bugs couldn't be found especially if you can make your PCIe device speak not-quite-right PCIe.

And since it's near impossible to validate FPGA firmware functionality by the kernel, rights to send bitstreams to the FPGA is essentially equivalent of root on DOM0.

[krupan]

Any PCIe device you plug into your computer has the same potential to do something nefarious. We already have problems where no two PCIe implementations interpret the spec the exact same way and they all have bugs. What you are hypothesizing isn't anything new.

[davrosthedalek]

The difference is that you don't need physical access anymore. You can convert a "good" device into a "bad" device via software.

[dooglius]

FPGAs are no different than any other hardware in this regard; in fact I suspect if you can hack the firmware on most pcie cards you could do that stuff too (unless there is an IOMMU)

[jgalt212]

It's more expensive, but at our next trade show, we'll move on from giving out hacked USB thumb drives, to hacked PCIe cards. Now I just need to figure out how get people to install them.

[LargoLasskhyfv]

Present it as NeuroPornSexcellerator for Midjourney, or something like that. Or maybe fast miner for cryptocoins.

[the_panopticon]

Yes. That’s one of the reasons we turn on the IOMMU even for boot https://cdrdv2-public.intel.com/671554/intel-whitepaper-usin...

[0xcde4c3db]

> Modern, high end FPGAs have a feature known as Raw SerDes, which in essence allows you to bypass a PCIe or Ethernet controller and use those lanes (yes, PCIe lanes) to your heart's desire ...provided you can design a working communication protocol. Difficult, but not impossible by any means.

Not even close to impossible. I've recently been trying to figure out a relatively "low-tech" way of talking to modern displays that doesn't involve feeding VGA into a black-box ADC, and from what I've gathered so far, most of the serial link standards developed in the past ~25 years are basically overclocked Fibre Channel with the serial numbers filed off and most of the reliability/ordering guarantees quietly dumped in a roadside ditch.

[throwaway2037]

> most of the serial link standards developed in the past ~25 years are basically overclocked Fibre Channel

Fascinating. Do you have a blog post about this?

[mips_r4300i]

What he means is that almost all serial interfaces at the PHY layer are derivatives of FC's 8b/10b coding. You turn bytes into symbols and optionally apply some whitening with an LFSR. Much like a radio TX. Then some additional tricks like de-emphasis where you try and overdue the high/low transitions to compensate for your PCB substrate or cabling being cheap.

PCI express, SATA, usb3, HDMI (in a slightly different way), display port, etc etc. all use some form of 8b10b coding (or more efficient coding) with multi-gigabit serial transceivers.

In fact the PHY layer for USB3, PCIe 2.0 and SATA is identical - Intel designed a physical standard to encompass all three of those called PIPE. Nowadays that only exists as a virtual bus between silicon IP blocks.

[0xcde4c3db]

Right. And not only is the general approach the same (which one could credibly blame on a sort of "parallel evolution" driven by the changing economics of transistors vs. pins in chip manufacturing), but in some cases the relevant standards literally incorporate parts of Fibre Channel by reference. It was seeing the DisplayPort spec cite the ANSI Fibre Channel PHY standard to define its specific 8b10b code (there have been several, due to the inherent redundancy of that type of code; cf. the choice of the 2 "extra" symbols in base64) that started me down the rabbit hole.

[wtallis]

I'm not sure how many similarities there are between current mainstream interfaces and Fibre Channel specifically, but there has been an obvious trend of convergence, and Fibre Channel was probably one of the earliest instances. Nowadays, almost all high-speed interfaces have abandoned parallel links in favor of serial signalling over differential pairs implementing a packet-switched network. There's enough similarity at the physical layer that you'll often see a chip designer go with a general-purpose PHY shared across SATA/SAS, USB, and PCIe ports because they all operate at similar speeds and often similar encodings (eg. 8b10b). Likewise for sharing between InfiniBand and Ethernet.

DRAM is still almost always a traditional parallel bus, and it's basically alone in that. HDMI was a late holdout where the signalling was serial on differential pairs but the clock speed was variable depending on the data rate; newer versions of the standard rely instead on the link operating at one of a handful of fixed standard data rates, as done by everything else.

[schmidtleonard]

> Modern, high end FPGAs have a feature known as Raw SerDes

That's like saying "did you know that advanced microprocessors have the capability to bypass I2C and output voltages directly on the pins?!?!?!?"

First of all, it's backwards. The physical layer comes before the protocols and is always there at the base. Second of all, the world does not exclusively run on I2C. Some people want SPI busses or to toggle transistors with GPIO. That's fine. Sure, gate it behind different permissions, but don't just rant at what you don't understand.

If you want a concrete example where serdes access is important, look up JESD204b, but in general there are loads of real-time systems or bespoke processing applications where it makes sense to dispense with the complex and temperamental packet-switched infrastructure in places where that complexity and nondeterministic behavior is likely to cause more trouble than good. There are also applications to backplane connections (if you are encapsulating PCIe, you want to run slightly faster than the PCIe so the PCIe can run at full bandwidth), even to the development of next-gen PCIe itself. It's not magic, it is not delivered by a stork, it needs to be prototyped, and that's another thing FPGAs are used for.

[justinclift]

Heh, you could do RDMA with other systems on the network using that approach too. ;)