This was supposed to be an easy-breezy write-up on how ZFS and jails facilitated the development of some code. Instead the journey took my favorite detour: learning things.
cloud-init is a standardized method for cross-platform cloud instance initialization developed by Canonical in Python. And because of some recent work (very cool!), "cross-platform" now includes FreeBSD. The dependency on Python means cloud-init cannot be included as a part of FreeBSD (a.k.a., "base"), and thus, the virtual machine images produced by the release engineering team won't be cloud-init enabled.
While cloud-init can perform a mind-boggling number of first-boot configuration management operations, my needs are more modest. Typically, the configuration tasks I need are:
- set the host name
- configure a network interface (IP address + subnet mask, default gateway, DNS)
- add an ssh key
The FreeBSD base does have a Lua interpreter, and there is an effort underway to implement a subset of the cloud-init operations in Lua. Getting this version into base would allow the VM images from release engineering to be cloud-init enabled for some use cases.
Initially, debugging the code took the normal route: create a VM image, snapshot it, add the current cloud-init bits, boot the image, and debug the result. Lather, rinse, repeat. But I quickly missed my development tools and wondered about using a jailed environment on my system. Since cloud-init focuses on "first boot configuration", being able to return the environment to an unadulterated state is important. By putting the jail on a ZFS dataset, you can create a snapshot (zfs snapshot pool/dataset@name) for the pristine system state, and then restore it (zfs restore pool/dataset@name). Here, my jail is "nuageinit", and the system image will be under the root/ directory.
# zfs create -o mountpoint=/jails zroot/jails
# zfs create -p zroot/jails/nuageinit/root
On to installing the bits in the jail. You would traditionally create the jail image by extracting the base.txz image, but package base allows us to create a smaller image using packages:
# mtree -deUW -f ~ctuffli/dev/freebsd/src.git/etc/mtree/BSD.root.dist -p root/
...
# mkdir -p /jails/nuageinit/root/usr/share/keys/pkg/trusted
# cp /usr/share/keys/pkg/trusted/* /jails/nuageinit/root/usr/share/keys/pkg/trusted/
# pkg -r /jails/nuageinit/root install -r FreeBSD-base \
FreeBSD-acct \
FreeBSD-newsyslog \
FreeBSD-runtime \
FreeBSD-rc \
FreeBSD-syslogd \
FreeBSD-utilities
How did I come up with the list of necessary packages? Trial and error. This is another case where ZFS snapshots shine.
Because we are debugging a script which modifies the system, we'll take a ZFS snapshot so that we can rollback the system state to try each new version of the code.
# zfs snapshot zroot/jails/nuageinit/root@freshinstall
The jail configuration file will perform the remainder of the workflow automation:
$j = "/jails";
$jd = "$j/$name";
$jp = "$jd/root";
$source = "~ctuffli/dev/ctuffli-nuageinit.git/libexec/nuageinit";
nuageinit {
devfs_ruleset = 14;
enforce_statfs = 1;
exec.clean;
exec.prepare = "mdconfig -a -t vnode -f $jd/seed.iso -u 1";
exec.created = "env DESTDIR=$jp BINDIR=/usr/libexec make -C $source install";
exec.created += "cp $source/../../rc.d/* $jp/etc/rc.d/";
exec.release = "mdconfig -d -u 1";
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = freebsd;
mount.devfs;
mount.fstab = $jd/fstab;
path = $jp;
allow.mount;
}
Most of the parameters are fairly standard. The interesting ones to note are the exec.prepare and exec.created values as these drive the workflow. cloud-init reads its configuration data from an attached disk volume labeled 'CIDATA' (well, for the NoCloud variant). The contents of the drive must use either a vfat or iso9660 filesystem. The "prepare" parameter takes the ISO image (seed.iso) with the cloud-init data and turns it into a block device passed into the jail (note the devfs_ruleset allows the jail to see the /dev/md1 device). The "created" parameter copies the latest version of cloud-init out of my home directory and installs the files in the jail.
The development workflow looks like:
- Hack on the code in my editor of choice
- Reset the system state in the jail:
# zfs rollback zroot/jails/nuageinit/root@freshinstall - Start the jail and get a console
# jail -c -f /jails/nuageinit/jail.conf
# jexec nuageinit /bin/sh - Start the service and test
- Shut everything down
# jail -r -f /jails/nuageinit/jail.conf nuageinit
[root@stargate /jails/nuageinit]# jexec nuageinit /bin/sh
# service nuageinit onestart
mount_cd9660: /dev/iso9660/cidata: Operation not permitted
/usr/libexec/flua: /usr/libexec/nuageinit:138: attempt to call a string value (local 'err')
stack traceback:
/usr/libexec/nuageinit:138: in main chunk
[C]: in ?
umount: /media/nuageinit: not a file system root directory
# ls -l /dev/iso9660/cidata
crw-r----- 1 root operator 0x320 Feb 12 23:04 /dev/iso9660/cidata
#Poop, that didn't work the way I wanted. While the jail can see the labeled image, mounting that images fails. The jail(8) documentation explains why:
allow.mount
privileged users inside the jail will be able to mount
and unmount file system types marked as jail-friendly.
The lsvfs(1) command can be used to find file system
types available for mount from within a jail. This
permission is effective only if enforce_statfs is set
to a value lower than 2.
Running lsvfs on the host shows:
# lsvfs cd9660 msdosfs tmpfs
Filesystem Num Refs Flags
-------------------------------- ---------- ----- ---------------
cd9660 0x000000bd 0 read-only
msdosfs 0x00000032 1
tmpfs 0x00000087 1 jail
Meaning, it is expected that mounting a cd9660 file system within a jail will fail. Asking on the mailing lists, the reason appears to be safety:
File systems where the kernel parses a binary disk image aren't generally
safe because a bad image can corrupt kernel state.
Digging into the service script, it
- Figures out if a cloud-init image exists
- Determines the cloud-init image type
- Mounts the image to /media/nuageinit
- Runs the script which does the configuration (/usr/libexec/nuageinit).
nuageinit {
...
exec.prepare = "mdconfig -a -t vnode -f $jd/seed.iso -u 1";
exec.prepare += "mkdir -p $jp/media/nuageinit";
exec.prepare += "mount_cd9660 /dev/md1 $jp/media/nuageinit";
...
exec.poststop = "umount $jp/media/nuageinit && rmdir $jp/media/nuageinit";
...
}
The service script will still fail, but we can manually run nuageinit:
[root@stargate /jails/nuageinit]# jail -c -f /jails/nuageinit/jail.conf
nuageinit: created
install -o root -g wheel -m 555 nuageinit /jails/nuageinit/root/usr/libexec/nuageinit
installing DIRS FILESDIR
install -d -m 0755 -o root -g wheel /jails/nuageinit/root/usr/share/flua
install -C -o root -g wheel -m 444 nuage.lua /jails/nuageinit/root/usr/share/flua/nuage.lua
install -C -o root -g wheel -m 444 yaml.lua /jails/nuageinit/root/usr/share/flua/yaml.lua
ELF ldconfig path: /lib /usr/lib
32-bit compatibility ldconfig path:
Clearing /tmp (X related).
Creating and/or trimming log files.
Updating motd:.
Updating /var/run/os-release done.
Starting syslogd.
/etc/rc: WARNING: failed to start syslogd
Starting cron.
Mon Feb 12 23:56:46 UTC 2024
[root@stargate /jails/nuageinit]# jexec nuageinit /bin/sh
# /usr/libexec/nuageinit /media/nuageinit/ nocloud
# cat /etc/rc.conf.d/hostname
hostname="mysupervm.example.com"
# ls -l /home/freebsd/.ssh/authorized_keys
-rw-r--r-- 1 freebsd freebsd 109 Feb 12 23:57 /home/freebsd/.ssh/authorized_keys
#
Fin
No comments:
Post a Comment