> I read on many different sites that AMD’s current CPUs beat Intel’s CPUs in terms of performance per watt. We can better achieve goals 2 and 3 (low noise and low power usage) by using fewer watts, so we’ll pick an AMD CPU and mainboard for this build.<p>Unfortunately many reviews are very misleading here. Zen 2/3 CPUs have good performance per Watt, that's true. But for a machine like this, which will be mostly idle, this is not the interesting metric and Zen 2/3 systems show that you can combine good perf/W with poor idle power consumption (which is not true for their monolithic APU brethren, which are used in laptops).<p>One of the biggest idle power hogs for these is the IO die, so make sure that XMP is disabled and the memory uses one of the slow JEDEC timings. This should be fine for a router. Check that the SoC/NB voltage is set to 1 V or less. Some boards set this higher. In the AMD CBS section of the board firmware there should be an item "SoC OC Mode" somewhere. Disable it. Some boards allow you to set a new PPT (package power target), but it's worth pointing out that values which are too low will make the CPU very slow because it essentially forces all cores to very low power states in order to meet the PPT since the CPU can't influence the baseline power (due to fabric and I/O die). The upside of using a reasonable PPT of e.g. 50-60 W is that you reduce power consumption if some errant task hogs the CPU.<p>These settings make a big difference, but only if the CPU is <i>really</i> idle. Even fairly light loads (e.g. on a desktop, moving the mouse on the background) has everything rev up. In deep idle (nothing running at all, no user interaction on a desktop) you might get a Zen 2/3 CPU down to around 20 Watts, but as soon as anything is happening at all we're straight back to the 40-70 W region.<p>Using an Intel system for this would have likely saved 10-20 W.
> Our network cards need PCIe 3.0, so that disqualifies 5 chipsets right away: only the A520, B550 and X570 chipsets remain.<p>No, the linked Wikipedia page only shows the PCIe lanes connected through the chipset and doesn't account for PCIe lanes provided directly from supported CPUs. X470 supports CPUs which have PCIe 3.0 and a board with x8/x8 mode like the ROG Strix X470-F <a href="https://www.asus.com/microsite/motherboard/AMD-X470/" rel="nofollow">https://www.asus.com/microsite/motherboard/AMD-X470/</a> should work just fine (and has no fan).
Very cool post. Thank you.<p>This article, and its many links, are helping noob me learn how to ask about I/O perf.<p>TechEmpower benchmarks HTTP servers. While contestants continue to improve, I've long been curious what the <i>theoretical</i> fastest HTTP server <i>could</i> be. <a href="https://www.techempower.com/benchmarks/" rel="nofollow">https://www.techempower.com/benchmarks/</a><p>Basically, an update to the "C10K problem" for the year 2021. <a href="https://en.wikipedia.org/wiki/C10k_problem" rel="nofollow">https://en.wikipedia.org/wiki/C10k_problem</a><p>A bit like how Daniel Lemire initially thought about parsing JSON. Paraphrasing: Why can't a JSON parser run at wire speed? "Parsing JSON quickly: early comparisons in the wild"
<a href="https://lemire.me/blog/2019/03/02/parsing-json-quickly-early-comparisons-in-the-wild/" rel="nofollow">https://lemire.me/blog/2019/03/02/parsing-json-quickly-early...</a><p>--<p>Also, I really appreciate including the sound and cooling considerations. It's just great seeing the process of system design accommodating (balancing) multiple goals. Bravo.
The most surprising details was [1] from init7.<p>>Compatibility requirements
There is no obligation on you to procure the hardware through us, and the hardware shown here is not the only possible hardware for you to use. There are also other compatible products, as long as the requisite «bi-di» fibre optic technology conforms to the following specifications (recommended: Flexoptix, more router information):<p>No Modem / ONT. Just a Router with compatible SFP optic. I wish more ISP do that. And not force me to use your crappy ONT or Wifi Router. They could of course go another router and provide actual decent ONT or WiFi Router. But the chance of happening, or they care about quality is slim.<p>[1] <a href="https://www.init7.net/en/internet/hardware/" rel="nofollow">https://www.init7.net/en/internet/hardware/</a>
I'm always amazed by people who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7. In this case replacing a typical router (which consumes like a lightbulb), with a full-fledged PC (probably consuming like x100 the power).<p>I guess some people around the world have quite cheap utility bills! For me, it's either a Raspberry Pi type of power consumption, or else a server that only powers on when needed. But I haven't learned yet how to do the latter, if possible at all.<p>In fact this is a nice place to ask: how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? Has this been tackled anywhere? There must be tons of people wanting a homeserver but living in places where electricity has a considerable cost...
What is the practical consumer (or even SME) use of a 10 or 25G connection?<p>As a consumer I’d definitely get it for the fun of it, but what is the point in reality?<p>We’ve just launched an ISP a couple of weeks ago here in Europe. Peak bandwidth use is about 3mbps per customer, and customers will download the same amount of data regardless of their plan.<p>I acknowledge this could be lack of imagination on my part, but the progression of bandwidth availability seems to be wildly outstripping demand (at least when it comes to fibre deployments).<p>Perhaps someone living in SF with a 25G connection can disabuse me of this notion.
If anyone is interested in building 10Gb router on a budget you can buy Mellanox OCP NIC with an adapter from OCP to PCIe for around 5~30 USD ( depend on your location )[0]. I recently build one 10G router with 4x10Gb, 2x1Gb ports for around 200 USD using second hand PC.<p>[0] <a href="https://www.youtube.com/watch?v=hkf3PkuKdOA" rel="nofollow">https://www.youtube.com/watch?v=hkf3PkuKdOA</a>
I spent a couple of hours last week trying to set up a fiber connection in our new office.
Since I didn't want to use consumer hardware in an office setting (and I already had a router that was compatible), I opted for getting my own hardware (a media converter).<p>I had _some_ prior knowledge of networking, but had never messed around with VLAN and MTUs before.
Luckily I found this[1] gist for a comparable setup that saved me.
I'm still only getting 350Mbps where I should be getting 1Gbps, but I _think_ that's just due to the bad networking chip in my cheap chromebook.
When I move all my stuff to the office, we'll see what we can top out at...<p>All this is to say, I definitely respect the effort the author put into this and it pains me to find out that my new SFP setup is already obsolete :).<p>[1]: <a href="https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef2627" rel="nofollow">https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef...</a>
One thing I've run into when building my own 10gbps router that I didn't see mentioned - you probably need to disable power saving / frequency scaling. The handling of packets is done in software, so when you start a download and the flood of initial data comes in at 10gbps, your CPU will still be running at 600 MHz and unable to keep up. It ramps up within 5-50 milliseconds depending on CPU, but for a TCP download the sender will have already interpreted the dropped packets as congestion and reduced their upload rate.<p>You can monitor this with ethtool stats to see how many packets the NIC dropped due to host buffers being full.
Was hoping for similar but 10 gbit. Anyone know a nice small/quiet linux box with 2x10 gbit for use as a router?<p>I have a 8 port x 1gbit ubiquiti router, but ubiquiti seems to be going downhill, and 8 port x 10gbit routers are pretty expensive. My plan was a 2 port router and use vlan tagging to a 8+ port 10g switch. That way I can have separate networks for trusted ethernet, trusted wifi, untrusted ethernet, and untrusted wifi.
Here I am having just had FTTH installed, but capped to a mere 160 Mbps. The main problem I am having is not the router (which handles such a "slow" speed fine) but the wiring throughout the house. It's all copper ethernet and because of the lengths and the fact that the runs must go parallel to some mains electricity cables, it tops out at 100 Mbps. I hope the poster has already put fibre around his house.
Kind of surprised there wasn't much attention paid to the software. My understanding is that with stuff like DPDK and fd.io you can get much better performance without going to extremes with your hardware. Netgate TNSR is one product that puts all the opensource pieces together to make it easy but it is all opensource software so should be usable by the average home user.
The real story here is how great the internet service can be now in a civilized country if you don't let Comcast have monopolies and ban local internet cooperatives and coast along selling the same service for 15 years while raising prices.<p>25Gbps symmetric for about $70/month. That's significantly less than I pay Comcast for its crappy 900Mbps down/40Mbps up service.
I have a Unifi UDMP, which while I think is probably the best prosumer option right now, falls way short due to a PPPoE problem that limits fiber connections to under 500mb/s, and the lack of load balancing across multiple link networks (which I totally admit, is a first world problem).<p>I am thinking about building my own, but then comes the maintaince of all of the hardware / software. For me that would be fine, but for my family, it would be a total PITA to manage, and any downtime would be horrible.<p>So, stuck with UDMP for the moment, until they either fix their problems, or alternatively I decide to bite the bullet and build this on top of linux.
This is not related to the article, but to networking in general: I just discovered Brocade switches and how cheap they can be for what you get.<p><a href="https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/" rel="nofollow">https://forums.servethehome.com/index.php?threads/brocade-ic...</a><p>Between wanting to dabble with a 10G file server and thinking of hosting 40-60 person LANs, this is incredible.
At some point the idea of "use more case fans, then all of them run at lower RPM" simply turns on you and instead produce more noise. For the low wattages of this build a single case fan would've been sufficient to keep PSU and CPU fan at silent levels.<p>I have a few more watts in my PC case, but not a single case fan, and the machine still hovers at a comfortably quiet level at 1 meter's distance. No temperature issues.
I really respect this build, the only thing that I don’t like is the power consumption at idle of ~50 Watt. I understand that this is probably due to the used NICs but still.<p>It might almost be interesting to setup a second router (a Pi4 might do) for regular casual internet usage + VRRP and only turn on this beast when more bandwidth is required.<p>The 25Gbit machine would be the VRRP master and the Pi4 the slave.
Not sure why you buy at fs.com when you have online stores in CH that are selling much cheaper:<p><a href="https://www.microspot.ch/de/computer-gaming/pc-komponenten/steckkarten--c589100/intel-netzwerkadapterkarte-pci-express-3-0-2-x-sfp28--p0001320718" rel="nofollow">https://www.microspot.ch/de/computer-gaming/pc-komponenten/s...</a>
To get a grip around the numbers, i quote from George Neville-Neil's Talk at BSDCan '15 "Measure Twice, Code Once" [1]:<p>- 10 Gbps is 14.8 million 64 byte packets per second
- 67.5 ns per packet or 200 cycles at 3 GHz
- Cache miss is 32 ns<p>[1] <a href="https://www.youtube.com/watch?v=LE4wMsP7zeA" rel="nofollow">https://www.youtube.com/watch?v=LE4wMsP7zeA</a>
Good post, bring me back old memories where I used to setup my own PCs back in high school.
I think the graphics card and GPU are really not required for a router. Few years ago there was a Linux project called zebra router which was deprecated in favor of <a href="https://www.quagga.net/" rel="nofollow">https://www.quagga.net/</a>
@secure: I'm a fan and love what you did with router7, gokrazy and your homelab. Note that this might be seen as "How I replaced my off-the-shelf 600CHF router with a 1700CHF custom PC router".<p>I hope you'll update this once Fiber7 updates their POP to show measured performance :-)
I have a very similar setup using a desktop as a router (opnsense, 2 WAN, fibre/switch) and it has a single, yet fatal flaw.<p>The power button. When you lose power, and the power comes back on, your router does not. If you are away, you must physically turn it back on.
I wish P4 fpga boards weren't so expensive for hobbyist. You can do fun stuff with them and create a custom network pipeline. It's totally overkill for a home network (they are used to create datacenter fabric) but it's nice and shiny :)
Without PPS measurements its hard to say how good this is. 1.488MPPS is needed to saturate a gigabit connection and that usually needs some tricks like poll mode drives/dpdk/etc on commodity hardware.
I’d strongly recommend people take a look at the supermicro (or your vendor of choice) Xeon-d or AMD embedded 3000 series.<p>Lower power draw, very quiet and more than powerful enough to push 25gbit with cpu left over for VMs.
Damn, $70 / month for 25Gbps symmetric; meanwhile, we're paying $130 / month for 1Gbps (400Mbps really) down / 40Mbps up in Comcast monopoly-land in PNW.