Skip to Content

Building and releasing

Build Rookery from a checkout, run the checks and the docs, and make a release.

The repository

FolderHolds
crates/rookery/rookery. On Linux the commands and rookery serve, in src/local/; on every system the screens
crates/rookery-page/The screens in a terminal, which do everything through the API of a server
crates/rookery-core/, rookery-api/, rookery-pty/, rookery-tui/Shared libraries. rookery-tui has the parts of the screens with no terminal in them
crates/vendor/The vendored tui-input crate
crates/rookeryd/, crates/box/rookeryd and box, built static for hosts
provision/ansible/The Rookery role
provision/bootstrap/First-boot provisioning
provision/bin/rookery.py (release, seeds, ISO), qemu.py, write-usb.sh
provision/config/example.json, the profile a new host starts from
plugins/The plugins that come with Rookery
container/The desktop image and Rookery's own image
ci/The images and scripts for the checks and the end-to-end tests
docs/This site
tests/Python tests, and the end-to-end tests in tests/e2e/

Build

rustup target add x86_64-unknown-linux-musl     # once
cargo build --release -p rookery
cargo install --locked --path crates/rookery       # into ~/.cargo/bin

The build first makes rookeryd and box as static musl binaries and places them in the role's files. It then packs the provisioning source and embeds it in rookery. It needs Python 3 for the packing.

A release build embeds release builds of rookeryd and box. A development or test build (cargo build, cargo test) embeds quicker ones:

PartProfileBuilt
rookeryd, boxhost-devOptimised for size, incrementally, with debug assertions
Rookery itselfdevLine numbers, no other debug information

For a debugger, CARGO_PROFILE_DEV_DEBUG=true gives Rookery's own crates full debug information.

While changing the role or plugins, point Rookery at the checkout instead of the built-in copy:

rookery config set source "$PWD"

Check

cargo test --locked
python3 -m venv .venv && .venv/bin/pip install -r provision/requirements.txt
.venv/bin/python -m unittest discover -s tests
ANSIBLE_CONFIG=provision/ansible.cfg .venv/bin/ansible-playbook -i localhost, provision/ansible/site.yml --syntax-check
vale README.md docs/content/docs

Two of the Rust tests compare the API that rookeryd and rookery serve describe with the committed documents in docs/openapi/. After changing a route, regenerate them. ROOKERY_UPDATE_DOCS=1 cargo test writes them.

In containers

With Docker or Podman you need nothing else on your machine. ci/run.sh builds an image with Rust and its targets, Python with Ansible, and Vale. Everything runs in it. CI runs the same script.

ci/run.sh check              # what CI checks
ci/run.sh check clippy vale  # some of it
ci/run.sh e2e                # the end-to-end tests
ci/run.sh shell              # a shell with the toolchain, the checkout at /src

The checkout is mounted, and files are written as you. Cargo's downloads and the target directory stay in the volumes rookery-ci-cargo and rookery-ci-target, so later runs are fast. ci/run.sh clean removes them and the images.

The end-to-end tests have two test hosts, studio and forge. Each is an Ubuntu container with SSH and an operator account, and the two run in parallel. A profile is applied to a host over SSH, with the role's tasks for the accounts, the plugins and rookeryd. The tests then use the host through Rookery's commands, rookeryd's API and rookery serve. Test cases that look at the same state of a host follow each other, so the state is applied once. The desktop packages, toolchains and browsers are left out. Some modules need less than that. e2e_access and e2e_controller.Controller use the bare host forge, which has only SSH. e2e_provisioners runs without a host. e2e_restore starts a second rookery serve on the host studio and leaves the host as it is. With Docker, ci/run.sh also starts an engine on the test network for the Docker plugin's cases. ROOKERY_CI_DOCKER switches it on or off, and ROOKERY_CI_DIND_IMAGES names images to load into it. The test hosts and the tests are on an internal network with no way out. Each run starts from new hosts and removes them afterwards. A run ends with the time of each test and the number of playbook runs. After a failure the hosts' logs are in build/ci/logs/.

To run one module or one test case, name it: ci/run.sh e2e e2e_serve.Roles. ci/run.sh e2e --shard plain runs one shard on a fresh host, as a CI job does. The test cases of each host are listed in order in tests/e2e/run.py, and cut into shards in SHARDS there. A new one goes after the ones whose state it uses, and into their shard.

Boot tests

ci/run.sh boot cloud                 # the cloud seed in QEMU
ci/run.sh boot iso                   # the USB installer on a blank virtual disk
ci/run.sh boot cloud --keep          # and leave the guest running afterwards
ci/run.sh boot cloud -- boot_host.Reboot

They need /dev/kvm. --tcg runs without it, but takes hours. QEMU runs in a container of its own, ci/boot.Dockerfile, which also holds the package cache, the release server and the tests. The guest's disks are files under build/boot/. No block device is ever attached. With --keep the run ends with the commands to log in over SSH or to reach the serial console.

SettingEffect
ROOKERY_CI_BOOT_ISOAn Ubuntu Server ISO already on your machine, used read-only. Without it the ISO is downloaded to build/boot/images
ROOKERY_CI_BOOT_ISO_SHA256The checksum it must have. Without it, Ubuntu's SHA256SUMS
ROOKERY_CI_BOOT_MEMORY, ROOKERY_CI_BOOT_CPUSThe guest's memory in MiB and its vCPUs, 4096 and 4 by default
ROOKERY_CI_BOOT_SSH_PORTThe loopback port of the guest's SSH with --keep, 22722 by default

The cloud image and the ISO are checked against Ubuntu's SHA256SUMS on every run. build/boot/KIND/report.json has the time of each step, the disk and memory used and how much came from the package cache. The modules are listed in order in tests/e2e/boot.py.

The release archive

python3 provision/bin/rookery.py release

writes build/rookery-release.tar.gz and build/release.sha256. The archive is deterministic for the same source files and uses an explicit allowlist: it never contains local host profiles, secrets, virtualenvs or build output. Host-specific installer images and seeds are private and must not be published.

The docs

cd docs
npm install
npm run dev         # http://localhost:3000
npm run build       # static site in docs/out

The guide pages are MDX files in docs/content/docs/(guide). The API reference is generated before every build from docs/openapi/ by docs/scripts/gather.mjs. The terminal screenshots in docs/public/screens/ are PNG files rendered with headless Chrome. They come from docs/scripts/screens.py, which runs the built binary against an invented set-up and a throw-away container. See the comment at its top, which also says how to render the PNG files again. Set DOCS_BASE_PATH when the site is served from a folder, such as /rookery on GitHub Pages.

Releases

GitHub Actions build on every push:

WorkflowDoes
CIThe checks above and the end-to-end tests, both with ci/run.sh, and Vale. Rookery is built once, and the tests of each test host run in a job of their own
BootThe boot tests. boot cloud runs for changes to provisioning. boot iso runs once a week and when started by hand, because the installer takes longest and is used least
DocsThe docs build, and publishing it to GitHub Pages from main
Buildrookery for each system as artifact rookery-<target>, the host programs as host-programs-x86_64-linux, and the container image ghcr.io/aiviarium/rookery, tagged edge from main

To release, push a tag such as v0.3.0, or create the release on GitHub's release page with a new tag. Build then:

  1. Builds rookery, rookeryd and box with the version of the tag, and checks that each says it. Runs boot cloud. When it fails, nothing is published.
  2. Publishes the release with the archives of rookery, their checksum files, install.sh, install.ps1 and VERSION. A release made on the page gets the files added.
  3. Pushes the image tagged 0.3.0, 0.3 and latest.
  4. Opens a pull request that sets the version on main in Cargo.toml, Cargo.lock and docs/openapi/. Merge it once its checks have passed.

You don't edit a version anywhere. A build from a checkout says the version in Cargo.toml, which is that of the latest release. That way rookery update never offers an older version. rookery, rookeryd and box take the version from rookery_api::VERSION. A release build sets it with ROOKERY_BUILD_VERSION, and docker build takes the same name as a build argument.

A tag with a suffix, such as v0.4.0-rc.1, makes a pre-release. Its programs say 0.4.0-rc.1 and its image is tagged only with that. main stays as it is. The install scripts and rookery update pass over pre-releases, and ROOKERY_VERSION=v0.4.0-rc.1 installs one.

On every system

rookery builds on Linux, macOS and Windows. Its installation, rookery serve and the commands are compiled for Linux alone, in crates/rookery/src/local/. On macOS and Windows it is the screens, rookery login and rookery update, a client of a server elsewhere. The program builds on the system it is for:

cargo build --release --locked -p rookery

A release builds it on a runner of each system and checks that it says its version:

SystemTargetArchive
Linux x86_64x86_64-unknown-linux-musl, with rookeryd and boxrookery-x86_64-linux.tar.gz
Linux arm64aarch64-unknown-linux-musl, with rookeryd and box for x86_64 hostsrookery-aarch64-linux.tar.gz
macOS on Apple siliconaarch64-apple-darwinrookery-aarch64-macos.tar.gz
macOS on Intelx86_64-apple-darwinrookery-x86_64-macos.tar.gz
Windows x86_64x86_64-pc-windows-msvc, with the C runtime linked inrookery-x86_64-windows.zip

On Windows, the build needs Rust from rustup and the Visual Studio build tools.

Edit on GitHub

Last updated on

On this page