Allow running tests with spread

Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.

Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.

The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.

Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:

```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```

This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.

Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase

This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.

I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.

A number of systems and tests are currently failing:

- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected

In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

.gitignore

@@ -306,3 +306,10 @@ tests/regression/apparmor/xattrs_profile
 tests/regression/apparmor/coredump
 **/__pycache__/
 *.orig
+# Patterns related to spread integration tests
+*.img
+*.iso
+*.log
+*.qcow2
+*.run
+.spread-reuse.yaml

spread.yaml

@@ -0,0 +1,182 @@
+project: apparmor
+backends:
+  google:
+    key: '$(HOST: echo "$SPREAD_GOOGLE_KEY")'
+    halt-timeout: 1h
+    # Run only when explicitly named. This backend requires a Google Compute
+    # Engine (GCE) account and incurs cost on every use. It is most practical
+    # to scale-out tests once spread can express sufficient concurrency.
+    manual: true
+    # TODO: This needs to be adjusted to properly account for apparmor tests.
+    location: snapd-spread/europe-west2-b
+    systems:
+      - ubuntu-22.04-64:
+          workers: 4
+      - ubuntu-24.04-64:
+          workers: 4
+      - ubuntu-24.10-64:
+          workers: 4
+  garden:
+    # The garden backend relies on https://gitlab.com/zygoon/image-garden
+    # TODO: Switch to a released version for better stability.
+    type: adhoc
+    # Use 2GB of RAM and four cores as otherwise we may not have enough memory
+    # to link the parser. It is better to have more workers than to have one
+    # big worker with lots of resources.
+    allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 4" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))"
+    discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS"
+    systems:
+      # All systems except for the one Ubuntu system are marked as manual.
+      # This way we don't accidentally spin up everything when someone runs
+      # spread without knowing better.
+      - opensuse-cloud-15.6:
+          username: opensuse
+          password: opensuse
+          workers: 2
+          manual: true  # Run only when explicitly named.
+          environment:
+            # openSUSE 15 ships very old default python.
+            PYTHON: /usr/bin/python3.11
+            PYTHON_CONFIG: /usr/bin/python3.11-config
+      - opensuse-cloud-tumbleweed:
+          username: opensuse
+          password: opensuse
+          workers: 2
+          manual: true
+      - debian-cloud-12:
+          username: debian
+          password: debian
+          workers: 2
+          manual: true
+      - debian-cloud-13:
+          username: debian
+          password: debian
+          workers: 2
+          manual: true
+      - ubuntu-cloud-22.04:
+          username: ubuntu
+          password: ubuntu
+          workers: 2
+          manual: true
+      - ubuntu-cloud-24.04:
+          username: ubuntu
+          password: ubuntu
+          workers: 2
+          manual: true
+      - ubuntu-cloud-24.10:
+          username: ubuntu
+          password: ubuntu
+          workers: 2
+
+exclude:
+  - .git
+  - "*.qcow2"
+  - "*.iso"
+  - "*.img"
+  - "*.log"
+  - "*.run"
+
+# Copy the project to this path on the test system.
+# This is also available as $SPREAD_PATH.
+path: /tmp/apparmor
+
+prepare: |
+  # Install build dependencies, depending on the type of system running.
+  case "$SPREAD_SYSTEM" in
+    debian-*|ubuntu-*)
+      apt-get update -qq
+      # TODO: extract this from README.md libapparmor section and unifiy with what is in .gitlab-ci.yml.
+      DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
+        attr \
+        autoconf \
+        autoconf-archive \
+        automake \
+        bison \
+        build-essential \
+        dejagnu \
+        flake8 \
+        flex \
+        gettext \
+        libdbus-1-dev \
+        libtool \
+        liburing-dev \
+        pkg-config \
+        python3-all-dev \
+        python3-gi \
+        python3-notify2 \
+        python3-psutil \
+        python3-setuptools \
+        python3-tk \
+        python3-ttkthemes \
+        swig
+    ;;
+    opensuse-*)
+      # On openSUSE the default gcc and python are very old. We can use more
+      # recent version of Python quite easily but perl extension module system
+      # does not want us to modify the CC that's baked into perl and all my
+      # attempts at using gcc-14 have failed.
+      zypper install -y \
+        attr \
+        autoconf \
+        autoconf-archive \
+        automake \
+        bison \
+        dbus-1-devel \
+        dejagnu \
+        flex \
+        gcc \
+        gcc-c++ \
+        gettext \
+        gobject-introspection \
+        libtool \
+        liburing2-devel \
+        make \
+        pkg-config \
+        python3-flake8 \
+        python3-notify2 \
+        python3-psutil \
+        python3-setuptools \
+        python3-setuptools \
+        python3-tk \
+        python311 \
+        python311-devel \
+        swig
+    ;;
+    *)
+      echo "Please add support for $SPREAD_SYSTEM to spread.yaml"
+      exit 1
+    ;;
+  esac
+
+  # TODO: add logic to skip this build phase and use prebuild binaries from
+  # GitLab pipeline. This should also reduce the number of dependencies we need
+  # to install above.
+
+  # Configure libapparmor. We have to pass CC and CXX explicitly if provided in
+  # the environment.
+  (
+    cd $SPREAD_PATH/libraries/libapparmor
+    sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python
+  )
+  # Build libapparmor.
+  make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)"
+  # Build apparmor_parser.
+  # The alternative builds sequentially to use less memory.
+  make -C $SPREAD_PATH/parser -j"$(nproc)"
+  # Build binary utilities (aa-exec and firends).
+  make -C $SPREAD_PATH/binutils -j"$(nproc)"
+  # Build python utilities.
+  make -C $SPREAD_PATH/utils -j"$(nproc)"
+
+# In case of failure, include the kernel version in the log.
+debug-each: |
+  uname -a
+
+suites:
+  tests/unit/:
+    summary: Unit tests that do not exercise the kernel layer.
+  tests/regression/:
+    summary: Regression tests for parser-kernel interaction.
+    prepare: |
+      # FIXME: `make -C tests/regression` does not do anything.
+      make -C "$SPREAD_PATH/tests/regression/apparmor" -j"$(nproc)"

tests/regression/apparmor/task.yaml

@@ -0,0 +1,86 @@
+summary: run all of the apparmor regression test suite
+# TODO: This is somewhat tedious and it'd be nicer if we could somehow generate
+# the variants without doing it "by hand" with the following one-liner:
+#
+# echo '$(foreach t,$(TESTS),$(info TEST/$t: 1))'| make -f Makefile -f /dev/stdin
+environment:
+  TEST/aa_exec: 1
+  TEST/access: 1
+  TEST/attach_disconnected: 1
+  TEST/at_secure: 1
+  TEST/introspect: 1
+  TEST/capabilities: 1
+  TEST/changeprofile: 1
+  TEST/onexec: 1
+  TEST/changehat: 1
+  TEST/changehat_fork: 1
+  TEST/changehat_misc: 1
+  TEST/chdir: 1
+  TEST/clone: 1
+  TEST/complain: 1
+  TEST/coredump: 1
+  TEST/deleted: 1
+  TEST/e2e: 1
+  TEST/environ: 1
+  TEST/exec: 1
+  TEST/exec_qual: 1
+  TEST/fchdir: 1
+  TEST/fd_inheritance: 1
+  TEST/fork: 1
+  TEST/i18n: 1
+  TEST/link: 1
+  TEST/link_subset: 1
+  TEST/mkdir: 1
+  TEST/mmap: 1
+  TEST/mount: 1
+  TEST/mult_mount: 1
+  TEST/named_pipe: 1
+  TEST/namespaces: 1
+  TEST/net_raw: 1
+  TEST/open: 1
+  TEST/openat: 1
+  TEST/pipe: 1
+  TEST/pivot_root: 1
+  TEST/posix_ipc: 1
+  TEST/ptrace: 1
+  TEST/pwrite: 1
+  TEST/query_label: 1
+  TEST/regex: 1
+  TEST/rename: 1
+  TEST/readdir: 1
+  TEST/rw: 1
+  TEST/socketpair: 1
+  TEST/swap: 1
+  TEST/sd_flags: 1
+  TEST/setattr: 1
+  TEST/symlink: 1
+  TEST/syscall: 1
+  TEST/sysv_ipc: 1
+  TEST/tcp: 1
+  TEST/unix_fd_server: 1
+  TEST/unix_socket_pathname: 1
+  TEST/unix_socket_abstract: 1
+  TEST/unix_socket_unnamed: 1
+  TEST/unix_socket_autobind: 1
+  TEST/unlink: 1
+  TEST/userns: 1
+  TEST/xattrs: 1
+  TEST/longpath: 1
+  TEST/nfs: 1
+  TEST/xattrs_profile: 1
+  TEST/dbus_eavesdrop: 1
+  TEST/dbus_message: 1
+  TEST/dbus_service: 1
+  TEST/dbus_unrequested_reply: 1
+  TEST/io_uring: 1
+  TEST/exec_stack: 1
+  TEST/aa_policy_cache: 1
+  TEST/nnp: 1
+  TEST/stackonexec: 1
+  TEST/stackprofile: 1
+execute: |
+  # Run the shell script that is named after the spread variant we are running
+  # now. The makefile runs them all sequentially via the "alltests" goal. Here
+  # we can parallelize it through spread and also have a way to run precisely
+  # the test we want, especially for debugging.
+  bash "$SPREAD_VARIANT".sh

tests/unit/binutils/task.yaml

@@ -0,0 +1,3 @@
+summary: Run unit tests of AppArmor binary utilities
+execute: |
+  make -C "$SPREAD_PATH/binutils" check

tests/unit/libapparmor/task.yaml

@@ -0,0 +1,3 @@
+summary: Run unit tests of libapparmor
+execute: |
+  make -C "$SPREAD_PATH/libraries/libapparmor" check

tests/unit/parser-tst-caching/task.yaml

@@ -0,0 +1,3 @@
+summary: Run apparmor_parser caching test from parser/tst
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" caching

tests/unit/parser-tst-dirtest/task.yaml

@@ -0,0 +1,3 @@
+summary: Run apparmor_parser dirtest test from parser/tst
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" dirtest

tests/unit/parser-tst-equality/task.yaml

@@ -0,0 +1,7 @@
+summary: Run apparmor_parser tests from parser/tst
+# This test is particularly slow. Those values are aimed at running fast enough
+# on a Raspberry Pi 5, representing a slower-ish computer.
+warn-timeout: 20m
+kill-timeout: 30m
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" equality

tests/unit/parser-tst-error-output/task.yaml

@@ -0,0 +1,3 @@
+summary: Run apparmor_parser error_output test from parser/tst
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" error_output

tests/unit/parser-tst-minimize/task.yaml

@@ -0,0 +1,3 @@
+summary: Run apparmor_parser minimize test from parser/tst
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" minimize

tests/unit/parser-tst-parser-sanity/task.yaml

@@ -0,0 +1,7 @@
+summary: Run apparmor_parser parser_sanity test from parser/tst
+# This test is particularly slow. Those values are aimed at running fast enough
+# on a Raspberry Pi 5, representing a slower-ish computer.
+warn-timeout: 20m
+kill-timeout: 30m
+execute: |
+  make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" parser_sanity

tests/unit/parser/task.yaml

@@ -0,0 +1,16 @@
+summary: Run apparmor_parser unit tests from parser/
+details: |
+  The parser has a number of different tests. Those are all represented as
+  spread task variants so that they are directly visisble and runnable.
+environment:
+  TEST/tst_regex: 1
+  TEST/tst_misc: 1
+  TEST/tst_symtab: 1
+  TEST/tst_variable: 1
+  TEST/tst_lib: 1
+prepare: |
+  # The test relies on make to build a binary.
+  make -C "$SPREAD_PATH/parser" -j"$(nproc)" "$SPREAD_VARIANT"
+execute: |
+  cd "$SPREAD_PATH"/parser
+  ./"$SPREAD_VARIANT"

tests/unit/utils/task.yaml

@@ -0,0 +1,7 @@
+summary: Run unit tests of python utilities
+# FIXME: Exclude openSUSE as the test depends on python3-ttkthemes
+# which are not packaged in the distribution.
+systems:
+ - -opensuse-*
+execute: |
+  make -C "$SPREAD_PATH/utils" check