[alexfoo]

I'm waiting for 3 DACs and a few other bits to arrive today to move closer to 10G networking at home. Moving house soon and the new place will have 2.5Gbps FTTP (both up and down) so I wanted to be prepared for that. Given my existing broadband is only 500/75Mbps FTTP I was fine with a 1GbE internal network and Wifi-6 meshing. I could have planned to move to 2.5GbE but it may have been a bottleneck at some point, so may as well push straight on to 10G.

I have a USW-Aggregation with 8 SFP+ ports arriving today too. Just have to install Intel X520-DA2 cards in two of my servers (Proxmox host and a general Linux server), and the NAS also has a 10G SFP+ port, and then connect it all up.

Most of it second hand from eBay for half the usual retail price.

[lostlogin]

Nice work, that agg switch is excelllwnt.

I went with some cheap eBay cards and slotted them into a synology and PC.

They work great and have for years.

https://www.ebay.com/itm/384094168784?_skw=connectx+mellanox...

[vermilingua]

I really have to wonder what can you use 10G for? I have 500M down from my ISP, and it is faster than I can imagine ever needing, unless I get into data-hoarding 8k movies.

[mwpmaybe]

My homelab has a 10G fabric (switched) for NFS, iSCSI, NVMe-OF, etc. and a 25G fabric (a mix of back-to-back and switched) for clustering (Ceph, DRDB, ZFS replication, migrating VMs).

I spun up some iSCSI-backed SQL Server a few months ago and 10G couldn't keep up with the workload, so I dropped in a pair of 100G ConnectX-4 cards with iSER (iSCSI Extensions for RDMA) support for that particular use-case.

Just because your uplink is less than 10G doesn't mean the rest of your network can't be a bit more capable. :)

[alexfoo]

True, I don't really feel limited by my existing 500Mbps down, but knowing I'll be having 2500Mbps up/down soon means I want to have the infra to handle it.

Basing things on 2.5GbE would certainly have been cheaper but some things don't support it (they either do 1GbE or 10G SFP+) so settling on 10G where possible made more sense to me. My future ISP also has a 5Gbps up/down option, but even I can't justify that right now.

My wife and kid just want their phones/laptops to work, and to be able to stream stuff to watch, they don't care about the underlying speed.

Having a faster network may make some of my work related things run a bit quicker. A few times a day I'll need to pull something big down (either an ISO or a bunch of docker images) and that can take up to 2 minutes with 500Mbps down. Having those take a fifth of that time will make it seem less of a roadblock to doing work. 2 minutes meant I went and got a cup of coffee and often got more distracted, 30 seconds should keep me at my desk and focused on what I was doing. That's not a big enough reason to justify it on its own obviously.

I also want to do offsite backups with/for various family members, so something better than 75Mbps up is going to be a huge boost. Getting 1Gbps+ out will be huge (assuming whatever is at the other end can support that).

I don't do any kind of data hoarding, I think I've got something under 4TB of data that I actually care about, and most of that are family photos/videos.

Deep down it's mostly because I'm a networking geek so it's fun to play with some new kit and make blinkenlights.

[Fnoord]

Going for a cup of coffee means physical walk. Detaching from focussed mode means your mind gets in diffused mode. This is where/when creativity ensues.

One thing to remember is 2.5 gbit/sec uplink is shared between all clients. So if one client is on 1 gbit, and one client could saturate their 1 gbit while switch and router can handle better. An advantage of that is QoS isn't needed to be applied manually.

So, for example, it maybe worth it to have higher than 1 gbit uplink on switch to router, and maybe a server to switch, but devices such as your TV or WLAN clients do not need such.

75 mbit up is pretty good compared to DSL (I bet it is cable), and yes 1 gbit up is nice for off-site backups. But the upsell of going above 1 gbit symmetric IMO isn't there.

Cable providers know this. Which is why they sit below 1 gbit symmetric, at a level average subscribers are comfortable with.

[alexfoo]

> Going for a cup of coffee means physical walk. Detaching from focussed mode means your mind gets in diffused mode. This is where/when creativity ensues.

Sure, but I want to choose when I do it, not have it forced upon me.

> 75 mbit up is pretty good compared to DSL (I bet it is cable)

It is FTTP not DSL or cable. BT Fibre 500 in the UK. Almost all of the deals through the legacy/monopoly provider (BT/Openreach) are asymmetric like this.

The 2500/2500 at the new property is a different provider that has their own network and so isn't tied into reselling Openreach's GPON infra.

[Fnoord]

Asymmetric fiber, ridiculous. That changes everything TBH. I didn't expect that to exist in 2026.

[alexfoo]

https://communityfibre.co.uk/fibre-deals for reference

[EDIT] The asymmetric supplier is BT via Openreach. Google something like "BT Fibre 500".

[zamadatix]

It's less "what new thing can you do" and more "what things involve noticeably waiting, how long is the waiting, and what else is impacted". E.g. updating a game on Steam practically takes slightly under half the time for me (1.2 Gbps actual rate) and has absolutely 0 impact to any other traffic in the house. If it was 10x the price to get 10x the bandwidth I wouldn't bother but it was actually about the same as my old cable modem plan.

[whalesalad]

What if you want to access your NAS at 10g+ speed? You're focusing on WAN when there is also the LAN side.

[simulator5g]

It just makes everything feel faster. I went from 500m to 2.5g thinking I would immediately go back (I really just wanted the upgrade to XGS-PON to run my own network) and then I couldn't go back. Its very much like using a higher refresh rate or a faster CPU...

[hedgehog]

I went from one dev machine to two at my desk so I connected them via 25GBe. With about 2.8GBps TCP throughput and RDMA available I don't have to think too much about task placement or cross-traffic. (specific hardware: Mellanox ConnectX 4 LX cards + a DAC cable).

[hylaride]

For most people, 500M is probably fine. But once you have a few family members, each streaming 4K movies to their devices, and a parent that needs a video call to work seamlessly, you start to see the benefits.

10G is probably overkill, but it's also future proofing. The way things are going, loading the NYtimes will require 10G just for the advertising alone...

[jech]

> For most people, 500M is probably fine. But once you have a few family members, each streaming 4K movies to their devices,

You must have a very large family. To saturate 500Mbit/s, you'd need around 30 family members all streaming at the same time.

[ahoka]

A 4k stream needs around 25M and video calls are more about QOS.

[Filligree]

How many PCIe lanes are you allocating?

The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.

I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…

Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.

[zamadatix]

PCIe is also a full duplex connection so 2x10G is still just 20G instead of 40G. For PCIe 2.0 an x8 connection should get you full bandwidth on both ports simultaneously while x4 will fall just short for simultaneously usage (but still higher than 1 port). Unless you're really hankering for that full 20G, in which case a 25G NIC is definitely the better pick, that means you can just slot it in an x4 slot off the chipset on even a standard desktop PC.

Funnily enough, if you want a dirt cheap PCIe 3.0 based card the MCX353A-QCBT and MCX354A-QCBT give 1/2 ports of 40G QSFP+. They support QSFP+ to SFP+ adapters, so you can plug a 10G SFP+ into the QSFP+ port, but they don't support 4x10G breakout unfortunately. I ended up using the 2 port variant in both of my NASes - one port is 40G between the 2 for dirt cheap fast backups and the other is adapted to 10G to connect to the rest of the home network.

[mwpmaybe]

I've started buying Intel E810s for most purposes, even for 10G links. (SFP28 ports are generally backward-compatible with SFP+ DACs and transceivers.) The ones you can get on eBay for cheap typically run Dell firmware but it's serviceable. An E810-XXVDA2 is Gen4 x8; as long as the host slot can physically accept the card connector you only need Gen4 x1 electrical for a single 10G link or Gen4 x2 for dual 10G or single 25G.

[alexfoo]

I'm only planning on using one of the SFP+ ports on each of the cards, the dual port cards were just more common and cheaper on eBay.

The specs say they require PCIe v2.1 x8 lane.

My Proxmox server is quite old and has a Gigabyte GA-X79-UP4 mobo and has loads of spare PCI slots. One slot is taken up by a generic graphics card as the Mobo has no on-board graphics. (I think I went for this mobo because of the number of SATA ports, but it was over 10 years ago so not entirely sure.)

My general Linux server is newer and has an ASUS Prime H610M-A D4 mobo. Only two PCI slots (not used at the moment) and so the Intel X540-DA2 will use up the PCIe 4.0 x16 slot leaving just a PCIe 3.0 x1 slot. But that's fine as this machine is just a CPU (i7-13700), 64GB RAM and a 2TB NVMe. Sticking a good graphics card in it for GPU related fun had been on my list for years but I never got around to it, now the prices are just insane so I'll ignore that for now or something second hand falls into my lap.

[whalesalad]

I just went through the same process over the last few months. I had a USW agg and ran out of ports so now I have the big dog 24-port version. Mainly wanted L3 routing capability but it’s nice having more ports to lagg connections.

[alexfoo]

The limiting factor for me is that I'm renting so I can't put my own cabling in to the property. And with the new place there's no existing cabling, nor any conduits to run anything in, and chasing things into the walls/etc is going to be prohibited by the landlord or just too expensive if I'm only in this place for a year or two.

The spools of bend insensitive fibre are pretty cheap and very discreet so I'll probably have a couple of those running along skirting boards/etc in order to connect disparate areas of the house. (The ONT is ~15m away from where the majority of the equipment will live, that's the main bit I have to bridge.)

[simoncion]

> The spools of bend insensitive fibre are pretty cheap...

For the benefit of other folks reading, I'll note that even regular, boring OM4 bends okay. I have mine running along the outside of preexisting molding and baseboards. My runs get down to a 1/4" bend radius in places and it seems to work just fine. Bend-insensitive fiber is definitely useful, but it may not be required for the run one is planning.

Though, one thing that regular OM4 is not is discreet. That two-strand cabling in its aqua-colored jacket is quite distinctive.

[whalesalad]

I used to sneak cabling around the basement and pop it out of AC register vents when I was renting. I had one cable coming out the access panel to a bathtub which was conveniently in my office as I was next to the bathroom.

This vid comes to mind when you said bend insensitive fiber: https://www.youtube.com/watch?v=Z2FbzCyiNr4

[alexfoo]

Edwardian houses in the UK rarely have that level of access. No basement at all and I can't lift the carpets and floorboards to get to where I might be able to pass things through/around. No AC ducts. No coax to be able to use MoCA either.

But, yes, that video is exactly the kind of thing I had in mind for the bend insensitive fibre.

It all depends how I set things up (and I can't tell that until I've had more access to the property). The ONT and the rack with the USW-Aggregation switch are 10 yards apart, in terms of absolute distance, but probably 20 yards if you follow the walls/skirting-boards/etc.

The FTTP is presented as 2.5GbE Ethernet (apparently) so I can either:

a) put my Unifi Express 7 next to the ONT and then need a fibre run (something like https://uk.store.ui.com/uk/en/category/accessories-modules-f...) from the SFP+ port on the Express 7 to the USW-Aggregation in the rack.

However this will be sub-optimal in terms of Wifi and I'll probably need extra APs to cover all three floors and out into the back yard.

b) put my Unifi Express 7 in the hallway in the middle of the house (which should give me full Wifi coverage with no extra APs). This would mean a short (2m) DAC to connect it to the USW-Aggregation nearby, and I can use a 20m long flat/flexible Cat-6 Ethernet cable to go between the ONT and the Unifi Express 7.

[bombcar]

I did similar with the Mikrotik CRS305-1G-4S+IN and some surplus eBay gear. The nice thing is the NAS and my MacBook dock both have 10G and are connected - and it’s noticeable.

[alexfoo]

I had a big debate with myself whether to go Mikrotik or Unifi. Being EU based I really wanted to go Mikrotik but ended up with Unifi as I'd had more experience of it when helping out friends/neighbours.

Maybe my "last house" (i.e. the one we'll get to see us through to retirement and beyond) will be Mikrotik based. By then I'll probably want as little computing stuff as possible and will just sit in a comfy chair doing crosswords and sudoku with a pencil.

[simoncion]

If one wants to play around with the shell and other administrative tools, one can download a bootable x86 install image from [0] that works fine in qemu. I assume it'll work fine on any other major VM that can boot Linux and provide virtual NICs that work with in-tree drivers.

It's documented (but -IMO- poorly) that the default username is "admin" and the default password is the empty string. The brand-new-as-of-today docs site is at [1], the "older" docs site is at [2], and -as documented on the older docs site- you can get a PDF of the docs at [3].

If you ever find yourself with an entirely spare hour or two, fire up the VM just to play around with the interactive shell that they have built. I may not have worked with enough Enterprise Devices to have an informed opinion... but once I understood what the shell was telling me, I found its use of color to be helpful both when attempting to learn the basic syntax of the shell and as a reminder of what tokens are valid in which contexts. I've never worked with another interactive shell that has such nice syntax-and-data validity hinting.

[0] <https://mikrotik.com/download?architecture=x86>

[1] <https://manual.mikrotik.com/docs/introduction>

[2] <https://help.mikrotik.com/docs/>

[3] <https://box.mikrotik.com/d/df76f0d495284eb1b6a1/>

[bombcar]

Mikrotik is "quite low level" if you want it to be (it reminds me somewhat of old Cisco IOS) but it works great.

And even if you're a bit scared of manual configurations, the web GUI and Claude understand it pretty well.

[mwpmaybe]

I use UniFi for most of my home network so It Just Works™, but I've thought about mixing in Mikrotik for e.g. the compute rack so I can play around with 100G+ links and more esoteric stuff like VXLAN.

[justsomehnguy]

> Most of it second hand from eBay for half the usual retail price.

You were scammed. X520 is old enough to drive a car, the shop should pay you to get it off their hands.

[alexfoo]

Ha. I meant the rest of the equipment (USW-Aggregation, Unifi Pro Max 16, UNAS Pro, Unifi Express 7) was somewhere around half retail price.

I think I paid ~$15 for each X520-DA2 including postage.

[mwpmaybe]

I've started buying E810s even for 10G links. PCIe Gen4, lower power draw, RDMA support, generally backward-compatible with SFP+ DACs and transceivers, and relatively inexpensive. Not nearly as dirt-cheap as the X520s but not crazy expensive (last I looked, at least). As I gradually replace switches over the next few years I can start taking advantage of 25G.

[theatrus2]

This is the way. The 10G cards are ancient and hog all the PCIe lanes.

I’ve had the best luck with Mellanox ConnectX 4 or 5 cards. The 5 can happily run 25G on a modern lane constrained system.

[erincandescent]

I've seen lots of pretty terrible experiences with the i40e and newer Intel drivers.

For newer NICs than the X520s I'd probably grab a Connect-X card.

[mwpmaybe]

They've been stable for me but I'd be lying if I said I'd only heard good things.

Looks like prices on the E810-XXVDA2 have come up since the last time I looked while prices on the ConnectX-6 Lx have come down, so that'd be a good option!