For years, I’ve been running a debian-wheezy container using systemd-nspawn. It used to have my fileserver, mail server, accounting system (sql-ledger), etc. The idea is that I’d backup the entire container rootfs and could quickly have it up and running on any machine. About all I use it for any more is the accounting system, which will soon be deprecated as well if I can figure out how to do invoices in hledger. All expenses and bank account tracking are already in hledger – that is working pretty well. With the recent hard drive crash, I needed to get the accounting system up and running again – sure enough, restore a backup of the container to my other workstation, and with the following:

sudo systemd-nspawn -b -D /scratch3/debian-wheezy-saturn

It’s up and running again.

Press ^] three times within 1s to kill container.                                                            
INIT: version 2.88 booting                                                                                   
Using makefile-style concurrent boot in runlevel S.                                                          
udev requires hotplug support, not started ... failed!                                                       
Filesystem mounted on /dev/shm; setting up compatibility bind mount. ... (warning).
Please remove this mount from /etc/fstab; it is no longer needed, and it is preventing completion of the transition to /run/shm. ... (warning).
Setting the system clock.         
hwclock: Cannot access the Hardware Clock via any known method.
hwclock: Use the --debug option to see the details of our search for an access method.
Unable to set System Clock to: Tue Nov 1 01:52:40 UTC 2022 ... (warning).
Activating swap...done.                                                                                      
Cleaning up temporary files... /tmp.
Mount point '/run/host/os-release' does not exist. Skipping mount. ... (warning).
Mount point '/run/host/os-release' does not exist. Skipping mount. ... (warning).
Mount point '/proc/sys/kernel/random/boot_id' does not exist. Skipping mount. ... (warning).
Mount point '/proc/sys/kernel/random/boot_id' does not exist. Skipping mount. ... (warning).
Mount point '/proc/kmsg' does not exist. Skipping mount. ... (warning).
Activating lvm and md swap...done.            
Checking file systems...fsck from util-linux 2.20.1
Mounting local filesystems...done.                                                                                                                                                                                        
Activating swapfile swap...done.
Cleaning up temporary files....           
Setting kernel variables ...done.          
Configuring network interfaces...RTNETLINK answers: Operation not permitted
Failed to bring up lo.                  
Cleaning up temporary files....                                                                              
startpar: service(s) returned failure: udev ... failed! 
INIT: Entering runlevel: 2                                                                                   
Using makefile-style concurrent boot in runlevel 2.                                                          
dmesg: klogctl failed: Operation not permitted                                                               
Starting enhanced syslogd: rsyslogd.         
Starting web server: apache2apache2: apr_sockaddr_info_get() failed for mars
apache2: Could not reliably determine the server's fully qualified domain name, using for ServerName
Starting periodic command scheduler: cron.                                                                   
Starting OpenBSD Secure Shell server: sshd.                                                                  
Starting PostgreSQL 9.1 database server: main.                                                               
Starting IMAP/POP3 mail server: dovecot.                                         

One nice things about a system-nspawn is that it is just a file-system on disk, so it’s super easy to tweak something from the host system without mounting a COW disk image or something, which is what you’d need to do with QEMU, Virtualbox, etc. The fact that I can run a crusty old debian rootfs for 8+ years on a modern Linux system is pretty amazing. It really speaks to the stability of the kernel and system abstractions, for it would use host kernel.


I use systemd-nspawn to do some tests of full disk images which we use for production at $DAYJOB. One thing which was a little not-intuitive was properly configuring the GUIDs for each partition in the disk so that systemd-nspawn could properly find the root file system within a multi-partition disk image. Thankfully it’s actually quite well documented once I realized what was needed to be added to the disk images: Discoverable Partitions Specification