Building an energy efficient server

I've built a server that's both powerful and quite energy efficient. It has modern hardware running on less than 10 Watts!

In this chapter I've documented my reasoning around the build and each component. I also explain the tweaks I've done for my specific setup.

info

This post was originally part of my developer notes.

Hardware

Let's start with the hardware. What did I get and why?

In order to tell you why, I need to tell you what requirements I had.

Requirements

• Since it will be running Home Assistant, and basically be the brain of my home, it needs to be reliable.
• It also needs to run 24/7 without costing a fortune in electricity, so low power usage.
• I really wanted it to be able to run a few virtualised workloads for when I want to play around with stuff, so decent performance.
• My 19" rack is connected to the kitchen, so it needs to be quiet.

Research

I did a lot of research on what hardware to get and looked at many different options.

info

One of the main resources I used was The Big Efficient Server Topic (Part 3) on a Dutch tech website called Tweakers.net. It really is a great resource for anyone looking to build an energy efficient server.

Components

These are the components I ended up with. I've tried to describe my reasoning for each component, but if you have any questions please don't hesitate to drop a comment.

Main components

#	Component	Product	Reasoning
1	Enclosure	Inter-Tech IPC 2U-20248	2U, small enough to fit in my very small server rack
1	Motherboard	Fujitsu D3642-B	With a cost around €200 it's not exactly a budget board, but it's a modern and energy efficient mobo that's well worth the price.
1	CPU	Intel Core i7-9700K Tray (SRG15)	Supports virtualisation technologies like VT-x. 9th gen because it's more efficient than 10th gen, and especially 11~13th gen. I7 over I3 because it's more powerful, even with the TDP being limited
1	CPU Cooling	Noctua NH-L9i Brown	I love this brand. Absolutely silent.
1	Memory	64GB Crucial CT2K32G4DFD832A	Memory size has a relatively small effect on energy usage. 64GB for running virtual workloads without worries.
1	Drive bay	DeLOCK 47221 drive bay panel	I didn't want to go for a SASS backplane because it might need upgrading when disk sizes become larger. Also challenging to find SASS platforms that don't require much power. This is rather simple: 6 sata cables directly to the mobo.
2	Solid state drives	Samsung 870 Evo (MZ-77E4T0B/EU) 4TB	To be used as the Array in Unraid, which requires 1 Parity disk (which you can not use). Unraid is basically utilising a modern version of software RAID.
1	Power Supply	Mini-box PicoPSU-90 + 84W (Fortron) Adapter Power Kit	Power supplies become very inefficient when utilised below 20%. This PSU is tailored towards idle load, which is expected to be lower than 10W

Accessories

#	Component	Product	Reasoning
1	Rack rails	Inter-Tech 18" telescopic rails 350mm - 450mm	For fitting it in the server rack. I had to customise these to fit in my very small 60cm rack
2	Sata cables	3 x 7-pin SATA 3 / 6Gbit/s 30cm	Standard cables that look nice
1	Sata to Molex	zdyCGTime SATA 15 Pin to Dual 4 Pin Molex One to Two Power Cable	PicoPSU only has 1x SATA power and 1x Molex power. This cable splits the SATA power connector into two Molex connectors, both of which go into the drive bay. It powers all six potential SSDs
1	USB flash drive (OS)	Verbatim Metal Executive USB 3.0 flash drive 64 GB, USB flash drive	The operating system will run from this USB flash drive, simply because it uses less power than when using another disk

The build

Here's what the build looks like.

Efficiency tweaks

Measuring

To measure the power usage I used a calibrated Nous A1T.

I calibrated the plug using a multi meter and a 40W light bulb (resistive load).

First boot

First thing I did was boot it up with no additional settings. Not surprisingly, it crashed while booting into Unraid. Loading an OS is CPU heavy task and the energy usage spiked above the maximum the PSU could handle.

So the first tweak was to

• set the CPU TDP limit to 35W.

After this everything worked very smoothly. 🎉

Time for stats!

Stress testing

I ran a stress test with mprime to see what the maximum power usage was.

Source: https://www.mersenne.org/download/

# Download
wget https://www.mersenne.org/download/software/v30/30.8/p95v308b17.linux64.tar.gz

# Extract
mkdir mprime && tar -xf ./p95v308b17.linux64.tar.gz -C mprime

# Manual
./mprime/mprime --help

# Run
./mprime/mprime -t

For the visual people among us, here's what that looks like:

Initial stats

Now that we know both our idle and stress power usage, we can start tweaking.

But before that, time for the initial stats!

Idle low	Idle avg	Idle high		Stress low	Stress avg	Stress high
11W	13W	16W		54W	56W	62W

What's interesting is that even without any tweaks, the idle power usage of the server is already very low.

This has everything to do with the specific hardware that was chosen, especially the motherboard with its focus on energy efficiency and the power supply unit (PSU) not being so large that it becomes inefficient at low loads.

info

The server will be running 24/7, so the average idle power usage is the most important metric here.

note

The artificial load of the stress test is not representative of real world usage. Therefore, I will not be using the high values but at maximum the avg values when reconfiguring max TDP for the CPU.

BIOS tweaks

BIOS tweaks: Turn off unused devices

Firstly, the most obvious tweaks: turning off all the things we don't need.

• Audio: Disabled
• PCI Express Slot 1: Disabled
• PCI Express Slot 2: Disabled
• PCI Express Slot 3: Disabled
• PCI Express Slot 4: Disabled
• Serial port: Disabled

Great results so far!

Idle low	Idle avg	Idle high
8W	10W	13W

BIOS tweaks: Review all relevant settings

CPU

• Enhanced Intel SpeedStep Technology: Enabled (was Enabled)
• Turbo Boost: Disabled (was Enabled)
• C States: Enabled (was Enabled)
• Package C State Limit: Auto (was Auto)

And since there's some headroom left, I also increase the maximum power limit.

• CPU TDP Limit: 60W (was 35W)

USB

• USB 3.1 Gen 2 speed: Disabled (was Enabled)
• USB Port Control: Enable rear and internal ports (was Enable all ports)

NIC

• Wake on LAN: Disabled (was Enabled)

PCI

• Above 4G Decoding: Disabled (was Enabled)

Idle low	Idle avg	Idle high
7W	9.5W	12W

BIOS tweaks: Disable graphics

The default settings for my motherboard are as follows:

• Primary display: Auto
• Internal graphics: Auto

info

When using Internal graphics: Auto, the BIOS detects automatically if the internal graphics controller can be disabled.

This means that the next test is to pull out the monitor cable and see how it affects power usage. Unfortunately pulling out the cable didn't make any difference.

I then manually disabled the internal graphics controller, which did seem to make a small difference, but still too small to measure.

info

At this point the difference also becomes hard to measure, as the power usage can jump around a bit and the update interval of the Nous A1T is about 1 second.

I wasn't able to detect a significant difference in power usage with this tweak.

Software tweaks

Software tweaks can help out where hardware settings are not sufficient or not available. One of the main things to focus on is the CPU, or more specifically, the C-states.

C-states

info

C-states, also known as CPU states, are power-saving states for processors (CPUs) in modern computer systems. These states are designed to reduce the power consumption of the CPU when it is idle or not performing any computational tasks.

The different C-states (C0 through C10) represent varying levels of power-saving modes:

• C0 is the operating state.
• C10 is the deepest idle state.
• The higher the C-state, the more power is saved.

We'll use Powertop to try and improve the C-state usage.

Powertop

info

Powertop is a tool provided by Intel to enable various power saving modes in userspace, kernel and on hardware level.

Powertop comes as one of the recommended tools to tweak power usage on Linux. It's a tool that gives you quite a bit of insight into what's going on with your system. It also allows you to tweak some settings.

I've followed this guide to install it on Unraid. I'll describe some of the steps here.

Install

Download the file in /boot/extra, then reboot to automatically install it.

mkdir /boot/extra
cd /boot/extra
wget https://github.com/mgutt/unraid-packages/raw/main/6.11.0/powertop-2.15-x86_64-1.txz

Analyse

Run powertop without --auto-tune (for now) to get an overview of what's going on.

powertop

On the Idle stats tab it looks like much of the time is spent in the C2 state.

Looking at the Tunables tab: It seems like there are many things to tune still.

Tuning

Now that we have an overview of what's going on, let's try to tune some of the settings.

powertop --auto-tune

# or alternatively
powertop --quiet --auto-tune

note

This command will be undone after every reboot, unless you add it to the go file. Keep reading 🙂

After running this command, the Tunables tab looks like this:

The Idle stats tab looks a lot better now too: Instead of the CPU spending most of the time in the C2 state it's now spending most of the time in the C8 state.

Keep tuning after reboot

First of, let's test that the array still comes up after tuning.

In Unraid stop the array, then run the tuning command again.

powertop --quiet --auto-tune

Now start the array again. If the array comes up, you should be good to go.

To keep the tuning after reboot, add the tuning command to the go file like so:

echo "powertop --quiet --auto-tune" >> /boot/config/go

Finally, reboot and check if everything still works as expected.

reboot

Results

Et voilà! After applying tuning, the power usage is down to 7W on average. Awesome! 🎉

Idle low	Idle avg	Idle high
6W	7W	10W

Conclusion

And there we have it!

A powerful server that uses only 7W on average when idle. I'm really happy with the results.

Credits

I could not have done it without the awesome contributions people made to the Big Efficient Server Topic (Dutch).

Special thanks to mrmrmr in particular. 🙏🏻