Ok so maybe I’m not a genius but felt pretty accomplished doing an incredibly nerdy thing. Getting four Ethernet ports on a supermicro server board running Ubuntu to work bonded together over a Cisco sg200-50p switch.
What’s going on is a little complicated but here’s the quick summary
- Most devices have 1Gbps Ethernet. So the maximum bandwidth you can see is about 100MBps. That’s pretty good to a hard disk but today SSDs can run at 2GBps!
- In the old days everyone shared a single 1Gbps link when Ethernet was first invented it was a bus technology. Like a party line. In those days you literally lay a coax down and could tee off computers
- However people got clever and we went from a bus to a star configuration and the modern Ethernet switch was born. It has a magic chip inside which gives the full 1Gbps each way between any two machines. There are limits of course but even cheap consumer switches can handle 5Gbps aggregate across their 8 ports. The small business grade Cisco sg200-50p has aggregate bandwidth of 100Gbps. That’s right it is a full cross bar which means it can give you the full 1Gbps in any direction.
- Last trick is something called Link aggregation control protocol. (lacp). If you think about it even with all this bandwidth you have a problem if you have say a server will a zillion machines trying to access it. In the end it only has 1Gbps each way. The solution is to put multiple. Usually four nics in the server and then make them look like a single gigantic virtual adapter. This is also called bonding.
- What happens is that the network switch itself needs to be programmed and that’s the main difference between a consumer unmanaged switch and a soho managed one. You now get the joys of yet another network device with a password and a place for the bad guys to put ease dropping software. But if you brilliant you can login and click on link management. Figure which ports you need. And on Cisco at least make sure to click LAG before you add connectors.
- Then on you Ubuntu machine you go to the network manager and choose bonding. Add each of the Mac ids of the hardware networks cards. Make sure to click access if available and then delete the individual entries so you get just one logical network called bond0
- Final problem is that some workstations can run into the same problem. The. Just make sure to get a fancy motherboard with dual nics and play the same trick. Bond them.
And voila if you spend an afternoon doing this then you don’t have to spend $250 a nic for 10Gbps Ethernet bin stead you get a server with 4Gbps and some workstations either 2Gbps. And you get to write a blog post too!
