PXE Booting an ALIX Board with FreeBSD

I recently needed to install FreeBSD on an ALIX 2D13 board. I decided to try PXE booting the installer, then installing to the compact flash card. There are other ways to accomplish this, but I'd been looking for an excuse to play with PXE.

There are a few main components to getting PXE booting working. The target board needs to use DHCP to get some specific information about where to find the PXE bootloader and root filesystem it needs. Then it uses TFTP to get the PXE bootloader and NFS to mount the filesystem.

The system serving TFTP and NFS can be the same one running the DHCP server, or not. In my case, I used my laptop for everything because I can't customize the DHCP configuration on my DD-WRT router. I'll replace it with an OpenBSD router soon.

These are the detailed instructions for exactly how I made this work. I pieced them together from multiple incomplete or outdated sources.

NFS

Get a FreeBSD image.
```
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/ISO-IMAGES/10.1/FreeBSD10.1-RELEASE-i386-disc1.iso
```
I'm using the full install image rather than the bootonly version because the ALIX board isn't going to have internet access while I'm doing this.
Create a directory for the installation files. We'll be serving this over NFS later.
```
mkdir /usr/public/fbsd10
```

Mount the ISO image you just downloaded and copy the files into this directory.

mdconfig -a -t vnode -f FreeBSD-10.1-RELEASE-i386-disc1.iso  
mount_cd9669 /dev/md0 /mnt  
cp -R /mnt/ /usr/public/fbsd10  
umount /mnt  
mdconfig -d -u 0

Comment out the CD line in /usr/public/fbsd10/etc/fstab
```
#/dev/iso9660/10_1_RELEASE_I386_CD / cd9660 ro 0 0
```
Export the installation directory over NFS in /etc/exports by adding this line.
```
/usr/public/fbsd10 -ro
```
This gives anyone who can connect read-only access.
At this point, NFS is ready to go. Start it up.
```
service nfsd onestart
```
It will start mountd and rpcbind as dependencies. If you make any changes to /etc/exports then you must restart mountd

TFTP

The target board will use TFTP to fetch the pxeboot binary.

Create a directory to hold the TFTP files.
```
mkdir /usr/public/tftp
```
Enable TFTP in /etc/inetd.conf
```
tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/public/tftp
```
This line should already exist, you just need to uncomment it and adjust the path on the end.
Copy the pxeboot executable from the installation files to the tftp directory.
```
cp /usr/public/fbsd10/boot/pxeboot /usr/public/tftp
```
Start inetd.
```
service inetd onestart
```
Now TFTP is all set.

DHCPD

Install isc-dhcpd43-server.
```
pkg install isc-dhcpd43-server
```

Add something like the following configuration to /usr/local/etc/dhcpd.conf

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.2 10.0.0.3;
    option subnet-mask 255.255.255.0;
    option routers 10.0.0.1;
    option broadcast-address 10.0.0.255;
    option domain-name-servers 10.0.0.1;
    # TFTP server
    next-server 10.0.0.1;
    # PXE boot file
    filename "pxeboot";
    # root filesystem
    option root-path "/usr/public/fbsd10";
}

Add the following lines to /etc/rc/conf
```
ifconfig_re0="inet 10.0.0.1 netmask 0xffffff00"
dhcpd_ifaces="re0"
```
This sets our ethernet port (re0) with a static IP and tells dhcpd to use it.
Start dhcpd.
```
service isc-dhcpd onestart
```

Testing

Make sure you can grab pxeboot over tftp.
```
tftp localhost
_> get pxeboot
```
Look in /var/log/xferlog if something is wrong.
Make sure you can mount the NFS share.
```
mount -t nfs 127.0.0.1:/usr/public/fbsd10 /mnt
umount /mnt
```
Look in /var/log/messages if something is wrong. If you have any trouble, double check your /etc/exports syntax and restart mountd again. To list all your shares, use:
```
showmount -e localhost
```

PXE Booting

Time to PXE boot the ALIX board. Connect the laptop to the board with ethernet. Connect your null modem FTDI serial cable, run screen /dev/ttyU0 38400, and power it on. Press s while the BIOS is starting up to get the BIOS menu. Change the baud rate to 9600 and enable PXE boot. Save and exit.
Restart screen without the baud rate argument and watch it (hopefully) PXE boot. If you see gibberish, you don't have the baud rate right. If it can't find the DHCP server, you're probably using the wrong ethernet port.

Installing

Go through the FreeBSD installer like normal, installing to the CF card. Choose 'Guided UFS'.
- Make sure you don't install the ports tree or the system source. We want to spare the CF card as much as possible.
- Make sure the hostname is a FQDN -- I used alixbuild.lan. If you don't do this, it will try to determine it during boot, which will cause it to hang for a minute.
- Add a user if you like. Invite the user to wheel and operator groups. Only members of wheel can su to root.
After the installer runs, ask for a shell in the chroot. There are several changes we need to make.
Edit /etc/fstab and add the option noatime to the root filesystem. It should look like this:
```
# Device        Mountpoint      FStype  Options         Dump    Pass#
/dev/ada0p2     /               ufs     rw,noatime      1       1
/dev/ada0p3     none            swap    sw              0       0
```
If you don't disable atime, it will write to the CF card every time a file is accessed, which will significantly shorten its life.
Edit /etc/ttys and specify off for each ttyv line. Find the ttyu0 line and make sure it looks like this:
```
ttyu0   "/usr/libexec/getty std.9600"   vt100   onifconsole secure
```
The startup script /etc/rc reads this file during boot to determine where to listen for logins with getty and how many virtual consoles to spawn. Because this board has no VGA output and we won't be connecting a keyboard to it, we don't need the normal virtual terminals.

We also need to make sure that the first serial terminal is set properly. The getty setting should match the baud rate we're using. In this case, it's std.9600
Add the following lines to /boot/loader.conf
```
autoboot_delay="3"
beastie_disable="YES"
```
This disables the ACSII Beastie display and shortens the delay to something more appropriate for an embedded device.
Edit /etc/ssh/sshd_config and set PermitRootLogin to yes. This will allow us to ssh in as root, which is disabled by default.
Add the following lines to /etc/rc.conf
```
ifconfig_vr1="DHCP"
sshd_enable="YES"
```
This tells it to connect to a router on the vr1 interface and enables ssh. This way, if the serial port doesn't work for some reason, we may be able to ssh in and recover.

Booting Up

Reboot the system and disable PXE boot in the bootloader. It should successfully boot from the CF card.

At this point, you can connect the vr1 interface to your router (or the DHCP server still running on your laptop) and ssh in. You can also talk to it over the serial port if you like seeing text print at 9600 baud :)

Update your system with freebsd-update fetch install and reboot again.

Next Steps

Now that your system is up, you may need to build ports if you need custom flags set. I recommend creating a new NFS share on your laptop to store the ALIX board's ports tree. Then mount it from the board and grab a ports tree.

mkdir /usr/ports
mount -t nfs 10.0.0.1:/usr/public/ports /usr/ports
portsnap fetch extract

Now when you build something, the file storage happens on your laptop over NFS, which will make the CF card happy.

Conclusion

There were a lot of things to get right because I combined PXE booting with running FreeBSD on an embedded device, but it worked well. I got hung up with NFS refusing to mount for a while because I forgot to restart mountd or had the syntax in /etc/exports wrong. But it was pretty straightforward.

Textplain

Tutorials