Using FreeBSD jails to debug cloud-init

This was supposed to be an easy-breezy write-up on how ZFS and jails facilitated the development of some code. Instead the journey took my favorite detour: learning things.

cloud-init is a standardized method for cross-platform cloud instance initialization developed by Canonical in Python. And because of some recent work (very cool!), "cross-platform" now includes FreeBSD. The dependency on Python means cloud-init cannot be included as a part of FreeBSD (a.k.a., "base"), and thus, the virtual machine images produced by the release engineering team won't be cloud-init enabled.

While cloud-init can perform a mind-boggling number of first-boot configuration management operations, my needs are more modest. Typically, the configuration tasks I need are:

• set the host name
• configure a network interface (IP address + subnet mask, default gateway, DNS)
• add an ssh key

The FreeBSD base does have a Lua interpreter, and there is an effort underway to implement a subset of the cloud-init operations in Lua. Getting this version into base would allow the VM images from release engineering to be cloud-init enabled for some use cases.

Initially, debugging the code took the normal route: create a VM image, snapshot it, add the current cloud-init bits, boot the image, and debug the result. Lather, rinse, repeat. But I quickly missed my development tools and wondered about using a jailed environment on my system. Since cloud-init focuses on "first boot configuration", being able to return the environment to an unadulterated state is important. By putting the jail on a ZFS dataset, you can create a snapshot (zfs snapshot pool/dataset@name) for the pristine system state, and then restore it (zfs restore pool/dataset@name). Here, my jail is "nuageinit", and the system image will be under the root/ directory.

# zfs create -o mountpoint=/jails zroot/jails
# zfs create -p zroot/jails/nuageinit/root

On to installing the bits in the jail. You would traditionally create the jail image by extracting the base.txz image, but package base allows us to create a smaller image using packages:

# mtree -deUW -f ~ctuffli/dev/freebsd/src.git/etc/mtree/BSD.root.dist -p root/
...
# mkdir -p /jails/nuageinit/root/usr/share/keys/pkg/trusted
# cp /usr/share/keys/pkg/trusted/* /jails/nuageinit/root/usr/share/keys/pkg/trusted/
# pkg -r /jails/nuageinit/root install -r FreeBSD-base \
        FreeBSD-acct \
        FreeBSD-newsyslog \
        FreeBSD-runtime \
        FreeBSD-rc \
        FreeBSD-syslogd \
        FreeBSD-utilities

How did I come up with the list of necessary packages? Trial and error. This is another case where ZFS snapshots shine.

Because we are debugging a script which modifies the system, we'll take a ZFS snapshot so that we can rollback the system state to try each new version of the code.

# zfs snapshot zroot/jails/nuageinit/root@freshinstall

The jail configuration file will perform the remainder of the workflow automation:

$j = "/jails";
$jd = "$j/$name";
$jp = "$jd/root";
$source = "~ctuffli/dev/ctuffli-nuageinit.git/libexec/nuageinit";

nuageinit {
    devfs_ruleset = 14;
    enforce_statfs = 1;
    exec.clean;
    exec.prepare = "mdconfig -a -t vnode -f $jd/seed.iso -u 1";
    exec.created  = "env DESTDIR=$jp BINDIR=/usr/libexec make -C $source install";
    exec.created += "cp $source/../../rc.d/* $jp/etc/rc.d/";
    exec.release = "mdconfig -d -u 1";
    exec.start = '/bin/sh /etc/rc';
    exec.stop = '/bin/sh /etc/rc.shutdown';
    host.hostname = freebsd;
    mount.devfs;
    mount.fstab = $jd/fstab;
    path = $jp;
    allow.mount;
}

Most of the parameters are fairly standard. The interesting ones to note are the exec.prepare and exec.created values as these drive the workflow. cloud-init reads its configuration data from an attached disk volume labeled 'CIDATA' (well, for the NoCloud variant). The contents of the drive must use either a vfat or iso9660 filesystem. The "prepare" parameter takes the ISO image (seed.iso) with the cloud-init data and turns it into a block device passed into the jail (note the devfs_ruleset allows the jail to see the /dev/md1 device). The "created" parameter copies the latest version of cloud-init out of my home directory and installs the files in the jail.

The development workflow looks like:

1. Hack on the code in my editor of choice
2. Reset the system state in the jail:
   # zfs rollback zroot/jails/nuageinit/root@freshinstall
3. Start the jail and get a console
   # jail -c -f /jails/nuageinit/jail.conf
   # jexec nuageinit /bin/sh
4. Start the service and test
5. Shut everything down
   # jail -r -f /jails/nuageinit/jail.conf nuageinit

[root@stargate /jails/nuageinit]# jexec nuageinit /bin/sh
# service nuageinit onestart
mount_cd9660: /dev/iso9660/cidata: Operation not permitted
/usr/libexec/flua: /usr/libexec/nuageinit:138: attempt to call a string value (local 'err')
stack traceback:
        /usr/libexec/nuageinit:138: in main chunk
        [C]: in ?
umount: /media/nuageinit: not a file system root directory
# ls -l /dev/iso9660/cidata
crw-r-----  1 root operator 0x320 Feb 12 23:04 /dev/iso9660/cidata
#

Poop, that didn't work the way I wanted. While the jail can see the labeled image, mounting that images fails. The jail(8) documentation explains why:

allow.mount
       privileged users	inside the jail	will be	able to	 mount
       and  unmount file system	types marked as	jail-friendly.
       The lsvfs(1) command can	be used	to  find  file	system
       types  available	 for  mount  from within a jail.  This
       permission is effective only if enforce_statfs  is  set
       to a value lower	than 2.

Running lsvfs on the host shows:

# lsvfs cd9660 msdosfs tmpfs
Filesystem                              Num  Refs  Flags
-------------------------------- ---------- -----  ---------------
cd9660                           0x000000bd     0  read-only
msdosfs                          0x00000032     1
tmpfs                            0x00000087     1  jail

Meaning, it is expected that mounting a cd9660 file system within a jail will fail. Asking on the mailing lists, the reason appears to be safety:

File systems where the kernel parses a binary disk image aren't generally
safe because a bad image can corrupt kernel state.

Digging into the service script, it

• Figures out if a cloud-init image exists
• Determines the cloud-init image type
• Mounts the image to /media/nuageinit 
• Runs the script which does the configuration (/usr/libexec/nuageinit).

Armed with this knowledge, we can modify the jail configuration to create this directory and mount the ISO in the jail.

nuageinit {
...
    exec.prepare  = "mdconfig -a -t vnode -f $jd/seed.iso -u 1";
    exec.prepare += "mkdir -p $jp/media/nuageinit";
    exec.prepare += "mount_cd9660 /dev/md1 $jp/media/nuageinit";
...
    exec.poststop = "umount $jp/media/nuageinit && rmdir $jp/media/nuageinit";
...
}

The service script will still fail, but we can manually run nuageinit:

[root@stargate /jails/nuageinit]# jail -c -f /jails/nuageinit/jail.conf
nuageinit: created
install  -o root  -g wheel -m 555  nuageinit  /jails/nuageinit/root/usr/libexec/nuageinit
installing DIRS FILESDIR
install  -d -m 0755 -o root  -g wheel  /jails/nuageinit/root/usr/share/flua
install -C  -o root  -g wheel -m 444  nuage.lua /jails/nuageinit/root/usr/share/flua/nuage.lua
install -C  -o root  -g wheel -m 444  yaml.lua /jails/nuageinit/root/usr/share/flua/yaml.lua
ELF ldconfig path: /lib /usr/lib
32-bit compatibility ldconfig path:
Clearing /tmp (X related).
Creating and/or trimming log files.
Updating motd:.
Updating /var/run/os-release done.
Starting syslogd.
/etc/rc: WARNING: failed to start syslogd
Starting cron.

Mon Feb 12 23:56:46 UTC 2024
[root@stargate /jails/nuageinit]# jexec nuageinit /bin/sh
# /usr/libexec/nuageinit /media/nuageinit/ nocloud
# cat /etc/rc.conf.d/hostname
hostname="mysupervm.example.com"
# ls -l /home/freebsd/.ssh/authorized_keys
-rw-r--r--  1 freebsd freebsd 109 Feb 12 23:57 /home/freebsd/.ssh/authorized_keys
#

Fin

Converting my init.vim to Lua

I've been using Vim and Neovim as my main editor for a bit (i.e., since the late 1990's) and have a small but useful-to-me configuration file. The Vim script configuration has served me well, but some newer Neovim functionality (e.g., the Language Server Protocol) uses Lua. Vim script allows this, but it requires using a here-document:

"""""""""""""""""""""""""""""""""""""""""""""""""
" Search recursively for files from "here"
set path +=**

"""""""""""""""""""""""""""""""""""""""""""""""""
" Set up Language Sever Protocol plugin

lua << EOF
-- Use an on_attach function to only map the following keys
-- after the language server attaches to the current buffer
local on_attach = function(client, bufnr)
  -- Enable completion triggered by 
  vim.api.nvim_buf_set_option(bufnr, 'omnifunc', 'v:lua.vim.lsp.omnifunc')

While this isn't terrible, it is a bit ugly and evidently, enough so to motivate converting my configuration to Lua.

Plugin Manager

My plugin needs have been modest (cscope bindings and mark) and haven't required a manager. But at some point, mark added a dependency on another package, making having a package manager more useful. vim-plug was a good match and has served me well, and while I'm changing over the init script to Lua, it made sense to use the packer.nvim package manager. Additionally, I moved all plugin definitions to nvim/lua/plugins.lua. This file also includes code to bootstrap packer if needed. After copying the files to a new machine, I use the command

$ nvim --headless -c 'autocmd User PackerComplete quitall' -c 'PackerSync'

to run the bootstrap and install the plugins. The plugins.lua file looks like:

local ensure_packer = function()
  local fn = vim.fn
  local install_path = fn.stdpath('data')..'/site/pack/packer/start/packer.nvim'
  if fn.empty(fn.glob(install_path)) > 0 then
    fn.system({'git', 'clone', '--depth', '1', 'https://github.com/wbthomason/packer.nvim', install_path})
    vim.cmd [[packadd packer.nvim]]
    return true
  end
  return false
end

local packer_bootstrap = ensure_packer()

return require('packer').startup(function(use)
    -- Packer can manage itself
    use 'wbthomason/packer.nvim'
    use {
        'inkarkat/vim-mark',
        requires = {'inkarkat/vim-ingo-library'}
    }

    -- Automatically set up your configuration after cloning packer.nvim
    -- Put this at the end after all plugins
    if packer_bootstrap then
        require('packer').sync()
    end
end)

Rosetta Stone

The Set List

The majority of the init.vim consists of set foo lines. This translates in a (mostly) straightforward way, except settings prefixed with "no", drop the "no" and use a value of "false". For example,

set nobackup		" do not keep a backup file, use versions instead

becomes

vim.opt.backup = false	-- do not keep a backup file, use versions instead

and

set title 		" change the window title to be the file name

becomes

vim.opt.title = true	-- change the window title to be the file name

Map to the stars

The configuration has a small number of key mappings. These also translate easily once you understand the syntactic sugar of Vim script. For example,

imap <S-Tab> <Esc> < i

is a mapping while in insert mode. Or

vnoremap  "*ygv

is a non-recursive mapping in visual mode. Armed with this knowledge, filling out the fields in nvim_set_keymap() is straightforward. Our cases above become:

vim.api.nvim_set_keymap("i", " ", "< i", {})
vim.api.nvim_set_keymap("v", "", '"*ygv', { noremap = true }) 

Some examples I found use keymap.set instead of nvim_set_keymap. The later appears to be an older version of the API, but I have chosen to use it as some Linux distributions ship older versions of Neovim which don't support the new syntax.

Bits-n-bobs

There were two other settings that don't fall into the "set" or "map" categories. The first allows recursively searching the current file path. Vim script uses += while Lua uses :append:

"""""""""""""""""""""""""""""""""""""""""""""""""
" Search recursively for files from "here"
set path +=**
" and search the Mercurial directory above if it exists
set path +=../.hg**

becomes

--------------------------------------------------
-- Search recursively for files from "here"
vim.opt.path:append('**')
-- and search the Mercurial directory above if it exists
vim.opt.path:append('../.hg**')

The other restores the cursor to its previous position in the file (see "BufWinEnter" below). The function mapping is as one would expect. The careful observer will notice a few settings disappeared. They are ones that either a) didn't get used or b) are holdovers from Vim.

Overall, this exercise was not too bad. See below for the before and after.

init.vim

" neovim settings
" relies on neovim defaults which are different than vim

set nobackup		" do not keep a backup file, use versions instead
set nowritebackup	" dont want a backup file while editing
set ruler		" show the cursor position all the time
set showcmd		" display incomplete commands
set title 		" change the window title to be the file name
set modeline
set cscopetagorder=0	" search cscope database before tags
set cscopetag		" search cscope database for tags
set nocscopeverbose
set termguicolors
set guicursor=
set background=light
set mouse=a

"""""""""""""""""""""""""""""""""""""""""""""""""
" Plug
call plug#begin()

" Dependency of vim-mark
Plug 'inkarkat/vim-ingo-library'
Plug 'inkarkat/vim-mark'

call plug#end()

" use ESC to close a terminal window
tnoremap <ESC> <C-\><C-n><C-w><C-p>

set clipboard+=unnamedplus
" From the neovim wiki under "clipboard=autoselect is not implemented yet"
" oh, and xsel doesn't seem to work but xclip does
vnoremap <LeftRelease> "*ygv

"""""""""""""""""""""""""""""""""""""""""""""""""
" Use all available colors for highlight (mark.vim)
let g:mwDefaultHighlightingPalette = 'maximum'

"""""""""""""""""""""""""""""""""""""""""""""""""
" Don't use Ex mode, use Q for formatting
map Q gq

"""""""""""""""""""""""""""""""""""""""""""""""""
" allow tab/shift-tab of selected sections
map <Tab> >0
map <S-Tab> <0
imap <S-Tab> <Esc> < i

" CTRL-U in insert mode deletes a lot.  Use CTRL-G u to first break undo,
" so that you can undo CTRL-U after inserting a line break.
inoremap <C-U> <C-G>u<C-U>

"""""""""""""""""""""""""""""""""""""""""""""""""
" Restore cursor to previous position in a file
function! ResCur()
	if line("'\"") <= line("$")
		normal! g`"
		return 1
	endif
endfunction

augroup resCur
	autocmd!
	autocmd BufWinEnter * call ResCur()
augroup END


"""""""""""""""""""""""""""""""""""""""""""""""""
" Convenient command to see the difference between the current buffer and the
" file it was loaded from, thus the changes you made.
" Only define it when not defined already.
if !exists(":DiffOrig")
	command DiffOrig vert new | set bt=nofile | r ++edit # | 0d_ | diffthis
				\ | wincmd p | diffthis
endif


" don't wrap long lines, but set the bottom scroll bar
" TODO this might be gvim specific
set nowrap guioptions+=b

"""""""""""""""""""""""""""""""""""""""""""""""""
" Search recursively for files from "here"
set path +=**
" and search the Mercurial directory above if it exists
set path +=../.hg**

init.lua

-- neovim settings
-- relies on neovim defaults which are different than vim

vim.opt.backup = false		-- do not keep a backup file, use versions instead
vim.opt.writebackup = false	-- dont want a backup file while editing
vim.opt.ruler = true		-- show the cursor position all the time
vim.opt.showcmd = true		-- display incomplete commands
vim.opt.title = true		-- change the window title to be the file name
vim.opt.modeline = true
vim.opt.cscopetagorder = 0	-- search cscope database before tags
vim.opt.cscopetag = true	-- search cscope database for tags
vim.opt.cscopeverbose = false
vim.opt.termguicolors = true
vim.opt.guicursor = ''
vim.opt.background = 'light'
vim.opt.mouse = 'a'

vim.opt.clipboard = 'unnamedplus'

--------------------------------------------------
-- Search recursively for files from "here"
vim.opt.path:append('**')
-- and search the Mercurial directory above if it exists
vim.opt.path:append('../.hg**')

--------------------------------------------------
-- Use all available colors for highlight (mark.vim)
vim.g.mwDefaultHighlightingPalette = 'maximum'

--------------------------------------------------
-- From the neovim wiki under "clipboard=autoselect is not implemented yet"
-- oh, and xsel doesn't seem to work but xclip does
-- TODO this seems to work with both nvim-0.6 and nvim-0.7 where as
-- vim.keymap.set("v", "<LeftRelease>", '"*ygv', {}) only works in 0.7
vim.api.nvim_set_keymap("v", "<LeftRelease>", '"*ygv', { noremap = true })

--------------------------------------------------
-- Don't use Ex mode, use Q for formatting
vim.api.nvim_set_keymap("", "Q", "gq", {})

--------------------------------------------------
-- allow tab/shift-tab of selected sections
vim.api.nvim_set_keymap("", "<Tab>", ">0", {})
vim.api.nvim_set_keymap("", "<S-Tab>", "<0", {})
vim.api.nvim_set_keymap("i", "<S-Tab> <Esc>", "< i", {})

--------------------------------------------------
-- Restore cursor to previous position in a file
vim.api.nvim_create_autocmd({ "BufWinEnter" }, {
    pattern = { "*" },
    callback = function()
        if vim.fn.line("'\"") <= vim.fn.line("$") then
            vim.api.nvim_exec('silent! normal! g`"zv', false)
        end
    end,
})

--------------------------------------------------
-- All Packer plugins in lua/plugins.lua
require('plugins')

Staring at mixed C/assembly code is way easier in Vim. Copy this to ~/.vim/syntax and then from the command line:

[674] objdump -S myprog | vim -c "setfiletype cmix" -

myprog:     file format elf32-i386
...
Disassembly of section .text:
...
08048504 <foo>:
#include <stdio.h>
#include <stdlib.h>

void foo()
{
 8048504:   55                      push   %ebp
 8048505:   89 e5                   mov    %esp,%ebp
 8048507:   83 ec 08                sub    $0x8,%esp
 804850a:   e8 c9 fe ff ff          call   80483d8 <mcount@plt>
    printf("Hello or Good-bye\n");
 804850f:   c7 04 24 50 86 04 08    movl $0x8048650,(%esp)
 8048516:   e8 cd fe ff ff          call 80483e8 <puts@plt>
}
 804851b:   c9                      leave  
 804851c:   c3                      ret    

0804851d <main>: