I've built a server that's both powerful and quite energy efficient. It has modern hardware running on less than 10 Watts!
In this chapter I've documented my reasoning around the build and each component. I also explain the tweaks I've done for my specific setup.
Let's start with the hardware. What did I get and why?
In order to tell you why, I need to tell you what requirements I had.
- Since it will be running Home Assistant, and basically be the brain of my home, it needs to be reliable.
- It also needs to run 24/7 without costing a fortune in electricity, so low power usage.
- I really wanted it to be able to run a few virtualised workloads for when I want to play around with stuff, so decent performance.
- My 19" rack is connected to the kitchen, so it needs to be quiet.
I did a lot of research on what hardware to get and looked at many different options.
These are the components I ended up with. I've tried to describe my reasoning for each component, but if you have any questions please don't hesitate to drop a comment.
|Inter-Tech IPC 2U-20248
|2U, small enough to fit in my very small server rack
|With a cost around €200 it's not exactly a budget board, but it's a modern and energy efficient mobo that's well worth the price.
|Intel Core i7-9700K Tray (SRG15)
|Supports virtualisation technologies like VT-x. 9th gen because it's more efficient than 10th gen, and especially 11~13th gen. I7 over I3 because it's more powerful, even with the TDP being limited
|Noctua NH-L9i Brown
|I love this brand. Absolutely silent.
|64GB Crucial CT2K32G4DFD832A
|Memory size has a relatively small effect on energy usage. 64GB for running virtual workloads without worries.
|DeLOCK 47221 drive bay panel
|I didn't want to go for a SASS backplane because it might need upgrading when disk sizes become larger. Also challenging to find SASS platforms that don't require much power. This is rather simple: 6 sata cables directly to the mobo.
|Solid state drives
|Samsung 870 Evo (MZ-77E4T0B/EU) 4TB
|To be used as the Array in Unraid, which requires 1 Parity disk (which you can not use). Unraid is basically utilising a modern version of software RAID.
|Mini-box PicoPSU-90 + 84W (Fortron) Adapter Power Kit
|Power supplies become very inefficient when utilised below 20%. This PSU is tailored towards idle load, which is expected to be lower than 10W
|Inter-Tech 18" telescopic rails 350mm - 450mm
|For fitting it in the server rack. I had to customise these to fit in my very small 60cm rack
|3 x 7-pin SATA 3 / 6Gbit/s 30cm
|Standard cables that look nice
|Sata to Molex
|zdyCGTime SATA 15 Pin to Dual 4 Pin Molex One to Two Power Cable
|PicoPSU only has 1x SATA power and 1x Molex power. This cable splits the SATA power connector into two Molex connectors, both of which go into the drive bay. It powers all six potential SSDs
|USB flash drive (OS)
|Verbatim Metal Executive USB 3.0 flash drive 64 GB, USB flash drive
|The operating system will run from this USB flash drive, simply because it uses less power than when using another disk
Here's what the build looks like.
To measure the power usage I used a calibrated Nous A1T.
I calibrated the plug using a multi meter and a 40W light bulb (resistive load).
First thing I did was boot it up with no additional settings. Not surprisingly, it crashed while booting into Unraid. Loading an OS is CPU heavy task and the energy usage spiked above the maximum the PSU could handle.
So the first tweak was to
- set the CPU TDP limit to 35W.
After this everything worked very smoothly. 🎉
Time for stats!
I ran a stress test with
mprime to see what the maximum power usage was.
mkdir mprime && tar -xf ./p95v308b17.linux64.tar.gz -C mprime
For the visual people among us, here's what that looks like:
Now that we know both our idle and stress power usage, we can start tweaking.
But before that, time for the initial stats!
What's interesting is that even without any tweaks, the idle power usage of the server is already very low.
This has everything to do with the specific hardware that was chosen, especially the motherboard with its focus on energy efficiency and the power supply unit (PSU) not being so large that it becomes inefficient at low loads.
The server will be running 24/7, so the average idle power usage is the most important metric here.
The artificial load of the stress test is not representative of real world usage. Therefore, I will
not be using the
high values but at maximum the
avg values when reconfiguring max TDP for the
BIOS tweaks: Turn off unused devices
Firstly, the most obvious tweaks: turning off all the things we don't need.
- Audio: Disabled
- PCI Express Slot 1: Disabled
- PCI Express Slot 2: Disabled
- PCI Express Slot 3: Disabled
- PCI Express Slot 4: Disabled
- Serial port: Disabled
Great results so far!
BIOS tweaks: Review all relevant settings
- Enhanced Intel SpeedStep Technology: Enabled (was Enabled)
- Turbo Boost: Disabled (was Enabled)
- C States: Enabled (was Enabled)
- Package C State Limit: Auto (was Auto)
And since there's some headroom left, I also increase the maximum power limit.
- CPU TDP Limit: 60W (was 35W)
- USB 3.1 Gen 2 speed: Disabled (was Enabled)
- USB Port Control: Enable rear and internal ports (was Enable all ports)
- Wake on LAN: Disabled (was Enabled)
- Above 4G Decoding: Disabled (was Enabled)
BIOS tweaks: Disable graphics
The default settings for my motherboard are as follows:
- Primary display: Auto
- Internal graphics: Auto
Internal graphics: Auto, the BIOS detects automatically if the internal graphics
controller can be disabled.
This means that the next test is to pull out the monitor cable and see how it affects power usage. Unfortunately pulling out the cable didn't make any difference.
I then manually disabled the internal graphics controller, which did seem to make a small difference, but still too small to measure.
At this point the difference also becomes hard to measure, as the power usage can jump around a bit and the update interval of the Nous A1T is about 1 second.
I wasn't able to detect a significant difference in power usage with this tweak.
Software tweaks can help out where hardware settings are not sufficient or not available. One of the main things to focus on is the CPU, or more specifically, the C-states.
C-states, also known as CPU states, are power-saving states for processors (CPUs) in modern computer systems. These states are designed to reduce the power consumption of the CPU when it is idle or not performing any computational tasks.
The different C-states (C0 through C10) represent varying levels of power-saving modes:
- C0 is the operating state.
- C10 is the deepest idle state.
- The higher the C-state, the more power is saved.
We'll use Powertop to try and improve the C-state usage.
Powertop is a tool provided by Intel to enable various power saving modes in userspace, kernel and on hardware level.
Powertop comes as one of the recommended tools to tweak power usage on Linux. It's a tool that gives you quite a bit of insight into what's going on with your system. It also allows you to tweak some settings.
I've followed this guide to install it on Unraid. I'll describe some of the steps here.
Download the file in /boot/extra, then reboot to automatically install it.
Run powertop without
--auto-tune (for now) to get an overview of what's going on.
Idle stats tab it looks like much of the time is spent in the C2 state.
Looking at the
Tunables tab: It seems like there are many things to tune still.
Now that we have an overview of what's going on, let's try to tune some of the settings.
# or alternatively
powertop --quiet --auto-tune
This command will be undone after every reboot, unless you add it to the
go file. Keep reading 🙂
After running this command, the
Tunables tab looks like this:
Idle stats tab looks a lot better now too: Instead of the CPU spending most of the time in the
C2 state it's now spending most of the time in the C8 state.
Keep tuning after reboot
First of, let's test that the array still comes up after tuning.
In Unraid stop the array, then run the tuning command again.
powertop --quiet --auto-tune
Now start the array again. If the array comes up, you should be good to go.
To keep the tuning after reboot, add the tuning command to the
go file like so:
echo "powertop --quiet --auto-tune" >> /boot/config/go
Finally, reboot and check if everything still works as expected.
Et voilà! After applying tuning, the power usage is down to 7W on average. Awesome! 🎉
And there we have it!
A powerful server that uses only 7W on average when idle. I'm really happy with the results.
I could not have done it without the awesome contributions people made to the Big Efficient Server Topic.
Special thanks to mrmrmr in particular. 🙏🏻