Hacker News new | ask | show | jobs
by eqvinox 1948 days ago
The only thing really atrocious on the RK3399 is the internal bus matrix. It's a legacy 32-bit crossconnect, i.e. physical address space is only 4GB.

Among other things, this means the PCIe aperture is squeezed down to a very measly 2×16MB — which is not even remotely enough for a lot of PCIe cards.

Also, if you install 4GB RAM, you only get to use something like 3.5G.

Oh and if I remember correctly, PCI MSI are broken, which again limits the PCIe card selection even more...
1 comments
CameronNemo 1948 days ago
Where can I find more information related to these issues?
link
eqvinox 1948 days ago
cat /proc/iomem on an installed Linux; note how everything is crammed into 32-bit addresses. Also,

http://opensource.rock-chips.com/images/e/ee/Rockchip_RK3399...

Page 13 ("1.1 Address Mapping") - same thing, everything crammed into 32bit.

PCIe Aperture is documented in TRM Part 2:

https://www.t-firefly.com/download/Firefly-RK3399/docs/TRM/R...

Page 711 - and I misremembered, it's 64MB total in 1×32MB + 32×1MB blocks. For comparison, my 6 year old server has 64GB PCI aperture space on each of its 2 sockets, my Laptop has 16GB. (Obviously both >32bit address spaces... which the RK3399 really should be too, but isn't.)

MSI is blacklisted in the Linux kernel somewhere if I remember correctly; it just randomly loses interrupts.

P.S.: Address space doesn't need to make a 32->64bit jump, my laptop CPU has 39bit, my server has 46bit; even 36bit (like old Intel 32bit PAE) would've been sufficient on the RK3399. But 32bit isn't.

P.P.S.: the PCI aperture size is the most obviously a problem when you try to plug in a GPU. E.g.:

  0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev cf) (prog-if 00 [VGA controller])
  ...
          Region 0: Memory at 900000000 (64-bit, prefetchable) [size=4G]
          Region 2: Memory at 880000000 (64-bit, prefetchable) [size=2M]
          Region 4: I/O ports at f000 [size=256]
          Region 5: Memory at f7e00000 (32-bit, non-prefetchable) [size=256K]
  ...
          Capabilities: [200 v1] Physical Resizable BAR
                  BAR 0: current size: 4GB, supported: 256MB 512MB 1GB 2GB 4GB
This GPU actually allows you to reduce the size it needs, but the absolute minimum you can set it to is 256MB...

The real problem is that PCI address space is not normally a scarce resource, and some RAID controllers & network cards (particularly high-performance/server ones) use a random bunch of it (>= 128MB) , and that's not even starting on SR-IOV. And these requirements aren't well documented because PCI memory space is not normally a problem. If you're lucky you can google "<card name> lspci", if not you only notice after you bought the card...

link