How to Install FreeBSD 12.0 on a PC Engines apu2 Machine (apu4c4)

Overview

This page briefly describes how to install FreeBSD 12.0-RELEASE on a PC Engines apu2, a small fanless x86-64 machine which can serve as a home NAT gateway router. The apu2 platform has some very attractive features, e.g., a serial console, plenty of Ethernet ports, and a variety of Mini PCI-e ports (see below). It is easy to run FreeBSD on once you know a few key facts which are laborious to discover on your own, hence this web page.

Ingredients

I bought an "assembled" apu4c4 system (power supply, case, motherboard and heat spreader installed in case) from mini-box.com. Due to PC Engines' unusual cooling design, many people may wish to pay to have a trained person marry the motherboard to the case. I picked the apu4c4 variant on the grounds that having a spare Ethernet port is better than running short.
You will pretty much need an mSATA "disk". At least as of FreeBSD 12.0-RELEASE and PC Engines BIOS v4.6.4, there seems to be a pretty thorough disagreement on the matter of SD cards. Using two different SD cards of different sizes and manufacturers (16 GB and 32 GB; Kingston and Sandisk), I was able to run the installer fine, but the system misbehaved badly (in different ways) when booting for real. So I affirmatively suggest you spring for an mSATA device. I happened to have a Kingston sms200s3/120g in stock, so that's what I used, but I suspect lots of devices will work equally well.
You will need a FreeBSD installer image on a memory stick. I chose FreeBSD-12.0-RELEASE-i386-memstick.img on the theory that the machine has only "32-bit infinity" of RAM, so there is no need to pay the 64-bit pointer-bloat tax for a machine that will just forward packets.
You will need a null-modem "DB9" serial cable. Note that you cannot tell by looking at a cable whether it is straight-through or null-modem, and it is not enough to have a DB9 serial cable with connectors of the right genders at both ends. You need to explicitly buy a null-modem cable, or a null-modem adaptor, or else use a multimeter to carefully check which pins go where. I am being so explicit about this due to having waded through a long FreeBSD forum thread where a wide variety of exotic theories were investigated and discarded, because the original poster thought that a serial cable with the right connectors at both ends must be the right kind of serial cable. But RS-232 is a land abundant in perfidy.
If you don't have a machine with an RS-232 serial port, you will likely need a USB-to-serial adaptor. I used a TRENDnet TU-S9. I'm sure other adaptors will work too.
You will need some sort of terminal emulator. I was using a laptop running Ubuntu 14.04 (the server variant), which apparently ships with a terminal emulator charmingly written in Python called miniterm.py. Obviously you can use something else, but you will need to know how to set the baud rate, manage character echo, etc., for whatever you use.
If you think you will want to have a Mini PCI-e wireless card, you might want to install it while you have the case open and are messing with the internals. You will likely also need RF pigtails with I-PEX connectors for the radio end, RP-SMA connectors for the antenna end, and assorted washers. Because you should avoid operating a wireless card without antennas, you will want those on hand as well. I used a Ubiquiti Networks SR71-E, which has the pleasant property of being a supported Atheros card. It has two antenna ports, matching the two antenna holes in the PC Engines apu4c4 case.

Steps

To run the FreeBSD installer over the serial console, you will want to make tweaks to the installer image before you get started. You will also wish to make tweaks so that the installer completes without hanging or crashing. This means you will need to mount the memstick image (perhaps in the form of a physical USB flash drive) from an existing FreeBSD machine (or virtual machine). Assuming the device shows up as /dev/da1s1a, you will want to take these steps.
1. mount /dev/da1s1a /mnt
2. echo '-h -S115200 -v' >> /mnt/boot.config
3. cat >> /mnt/boot/loader.conf << EOF # force serial console boot_serial="YES" console="comconsole" comconsole_speed="115200" # wait longer for USB-stick root device kern.cam.boot_delay="15000" kern.cam.scsi_delay="15000" # avoid igb netisr crashes hw.igb.enable_msix=0 hw.pci.enable_msix=0 EOF
4. Add these lines to /boot/device.hints, arguably at the top:
  # avoid fatal mSATA timeouts hint.ahci.0.msi="0" hint.ahci.1.msi="0"
5. umount /mnt
6. camcontrol eject da1
To install the mSATA device:
1. It goes in the "J13 mSATA/modem" slot, which is the one closest to the SD card slot and the clock battery.
2. Ground yourself (e.g., to one of the motherboard mounting screws), insert the card into the slot (the far end with the screw holes will be elevated compared to the slot end), and screw the card down.
If you are installing a wireless card:
1. It goes in the "J15 wifi" slot, the furthest away from the SD card slot and the clock battery.
2. You will likely want to follow the installation procedure recommended by PC Engines.
3. The apu4c4 case has two rubber plugs filling the antenna holes; they are easy to pop out. Note that the holes are only mostly round, with a flat side (the RP-SMA connectors on your RF pigtails probably match).
Connect up the serial cable and, on your other machine, start up your terminal emulator running at 115200 baud. For me it was:
miniterm.py /dev/ttyUSB0 115200
If you are using miniterm.py, you will want to hit ^T^L to disable wonky linefeed behavior.
Plug your memory stick into a USB port on the apu2 box (I used the bottom one due to having read allegations online that it might work better than the top port).
PC Engines suggests that you plug the power supply into the machine before you plug into your wall outlet, on the grounds that any arcing will happen in the wall outlet rather than sullying the connector on your apu4c4 motherboard.
You should hear from the BIOS basically as soon as you apply power:
```
coreboot build 20171130
BIOS version v4.6.4
4080 MB ECC DRAM
SeaBIOS (version rel-1.11.0.1-0-g90da88d)
Press F10 key now for boot menu
```
If you don't see something like that, check your cabling and your baud rate.
As directed, hit F10 so you can select your installer memstick.
You should see the boot block echo the line you put in /boot.config, then you should see the main boot loader load things, then you should see the "beastie" boot loader menu, the timer should tick down, then the kernel should be loaded, the kernel should chatter about hardware for a while, and then the installer should start.
I found that the "vt100" terminal type worked much better than the "ansi" type. Your mileage may vary depending on which terminal emulator you are using.
The installer will start off by asking you to set a hostname.
I stuck with the default list of optional components to install ("kernel-dbg").
For a partitioning approach, I chose "Auto (UFS)".
FreeBSD had no trouble finding the mSATA device, calling it /dev/ada0. I chose to install to ada0, "entire disk", MBR partitioning scheme. The partitioner suggested 3.8 GB of swap and 108 GB of /, which was fine by me. Alarmingly, the partition editor displays all partitions on all devices without a clear indication of which ones it intends to format/overwrite. But I didn't run into trouble.
The installer may appear to offer you various cool-sounding things, such as the ports tree, but if you are using a memstick installer those things probably aren't present, so you can save some time by not selecting things that don't exist.
The installer offers you a list of network interfaces to configure. As it turns out, it will configure exactly one interface (if you manually go back and do a second one it will lose the configuration you did for the first, so don't bother). On my apu4c4, igb3 is the one closest to the power plug, whereas igb0 is closest to the serial console port. Note that on multi-field installer screens you must be very careful to use the down-arrow and up-arrow keys to move between fields. If you accidentally press Return, you are done with that screen, and there is no simple way to get back. If you are familiar with the installer, you can find your way back later, but it is probably much better to just be very careful on the network-interface screen and the DNS-settings screen.
I told it that the system clock is set to UTC. For a single-OS platform (i.e., you are not dual-booting Windows) this is almost certainly what you want to pick.
For the "start at boot" settings (actually: /etc/rc.conf directives), I chose just "sshd" and "dumpdev". You might want to pick "ntpdate" and/or "ntp", though each will cause a fair amount of whining if you boot the system without a completely functional network setup, which you might be doing for a couple of boots.
For "system hardening" I picked:
- clean the /tmp filesystem on system startup
- disable opening Syslogd network socket
- disable Sendmail service
I added a user account (you may want to do this, since root logins via SSH are probably disabled; if you do this, you probably want to "invite" the non-root user to the "wheel" group).
When you hit the "exit installer" step (before you are offered a shell), don't be alarmed by a 30-second silent pause.
When you are offered the opportunity for a shell prompt, take it.
1. Run "date" and make any needed adjustments.
2. Manually re-do the changes you made to the installer's /boot.config, /boot/loader.conf, and /boot/device.hints, but this time you will use the real pathnames instead of ones involving /mnt, because you will be tweaking the system that will run after you are done with the installer. You can skip the "-v" on the /boot.config line this time if you want, because if you have finished the installer successfully you probably aren't running into any problems that a verbose boot would help you debug. Also, you can probably skip the "kern.cam.boot_delay" and "kern.cam.scsi_delay" lines in /etc/loader.conf.
3. Check /etc/fstab for "the da0/da1 disharmony". It is likely that your memstick installer ended up as /dev/da0. If, instead of using an mSATA device that showed up as ada0 as I suggested, you installed onto a second USB memory stick, that would likely have been called /dev/da1. If so, the installer would have generated an /etc/fstab referencing various slices of /dev/da1. When you remove the installer and reboot, the USB memory stick you installed to will be the only daX device, so it will be da0, but the mystical root determination process will try to mount /dev/da1s1a because that's what /etc/fstab says, and tears will result. If your /etc/fstab mentions /dev/da1 in any way, it is likely that you are having a bad problem and you will not go to space today.
Now it should be safe to exit the shell and accept the installer's suggestion to reboot. You will want to quickly yank the installer from the USB port as soon as you see the BIOS talking.
Hopefully the system will come roaring back up.

Adjustments, extensions, and tricks

I noticed that sshd ends up listening on *.ssh, which is less than ideal for a machine with two network interfaces, one of which will face the public Internet, so I edited /etc/rc.conf to set
```
sshd_flags="-4 -o'ListenAddress 192.168.1.1'"
```

You might want to make these additions to /boot/loader.conf:

# AMD processor temperature sensing
amdtemp_load="YES"
# Partial AMD crypto hardware support
aesni_load="YES"

I found that slowing the clock rate down to 600 MHz lowered the core temperature by 2 degrees Celsius, which might well extend the longevity of the hardware. If you want to do this, make a tweak to /etc/sysctl.conf:
```
dev.cpu.0.freq=600
```

Now you should be in business, perhaps interested in things like:

# /usr/sbin/pkg
# pkg update
# pkg search dhcp
# pkg install isc-dhcp44-server

If you are running ntpd, you will want an up-to-date leap-seconds file to avoid syslog messages like this:
```
ntpd[1141]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): expired less than 559 days ago
```
Or perhaps you don't want your system to be one second off every couple of years. There is a periodic(8) job designed to keep the leap-seconds file up to date. FreeBSD 12.0-RELEASE ships with that job enabled, but it will quietly fail due to an SSL certificate-authority problem. The fix:
```
# pkg install ca_root_nss
```
If you are a very careful person, you will spend a while carefully evaluating which certificate authorities to trust and will take the actions pointed out by the package installation message.

If you happen to want to temporarily bring up a wireless interface manually without configuration fuss/debugging:

# cat > /tmp/wpa_supplicant.conf << EOF
network={
       ssid="[insert SSID here]"
       psk="[insert password here]"
}
EOF
# ifconfig wlan0 create wlandev ath0
# wpa_supplicant -c /tmp/wpa_supplicant.conf -i wlan0
...wait a bit...
# dhclient wlan0
...wait a bit...

BIOS considerations

A newer BIOS might improve the negative behaviors that I observed with respect to SD cards, mSATA timeouts, and ibg-related crashes in the networking stack. On the other hand, I don't think I've found a definitive recommendation for a FreeBSD-friendly BIOS version newer than what I have. The space of BIOS versions is large and two-dimensional, and reports of the BIOS-flashing process going a little awry aren't rare, so I was more interested in getting the machine I had running reliably than in experimenting. Luckily these boxes are cheap enough that I can order a second one to serve as a backup and experimental subject.

Other efforts

After writing up this guide, I found several others... I had been searching for FreeBSD 12 installation instructions, and didn't realize there were probably FreeBSD 11 instructions floating around (1, 2) that I could have started from.