There's contention between running spread across many nodes, in chunks,
in a CI/CD pipeline, and running spread on one machine, across many
instances at the same time. The case with CI/CD needs one worker, as
parallelism is provided by GitLab. The case with local spread needs many
workers as parallelism is provided locally by spread allocating new
instances.
At present we need to focus on the CI/CD case. I have a plan on how to
avoid the problem entirely down the line, by running multiple copies of
spread locally, as if everything was done in a CI/CD pipeline.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
We now have GitLab CI/CD pipeline co-existing with spread, coupled with
image-garden and the cloud-init profile defined for each distribution.
To avoid duplicating list of required dependencies, re-use cloud-init
profile as the reference list of dependencies (superset between build
and test) to install.
In addition to the dependency list, the build_all job now re-uses spread
prepare section in similar fashion. If it builds in spread, it should
build in CI as well.
A small quality-of-life improvement is the shape of a collapsible
section around dependency installation should make reading job logs
easier.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1494
Approved-by: Christian Boltz <apparmor@cboltz.de>
Merged-by: Zygmunt Krynicki <me@zygoon.pl>
We now have GitLab CI/CD pipeline co-existing with spread, coupled with
image-garden and the cloud-init profile defined for each distribution.
To avoid duplicating list of required dependencies, re-use cloud-init
profile as the reference list of dependencies (superset between build
and test) to install.
In addition to the dependency list, the build_all job now re-uses spread
prepare section in similar fashion. If it builds in spread, it should
build in CI as well.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This is something that was done interactively as a part of a training
session.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1487
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
Tests that interact with the kernel are skipped (tests/regression and
tests/snapd) but everything else is green. Most of the tests are
actually passing. The only exception is the aa-notify test that was
broken by Python 3.13 stdlib change. The fix for that has been posted
separately.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1496
Approved-by: Christian Boltz <apparmor@cboltz.de>
Merged-by: Zygmunt Krynicki <me@zygoon.pl>
Tests that interact with the kernel are skipped (tests/regression and
tests/snapd) but everything else is green. Most of the tests are
actually passing. The only exception is the aa-notify test that was
broken by Python 3.13 stdlib change. The fix for that has been posted
separately.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Hopefully more and more profiles will come with smoke tests. Since the
pattern of those tests is likely to be very similar (compile profile,
run some programs, remove profile) it will be good to check if the
profile had caused any denials to be logged. Having this at the suite
level should make writing actual tests easier.
The prepare-each and restore-each logic compile the profile, check for
errors and finally remove the profile. The debug-each logic shows the
program name (with full path).
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Those fell under the radar during the initial push to expose all of
the tests to spread.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
By placing a bzImage into the top level of the AppArmor git repository one can
instruct spread and image-garden to use that image instead of booting
traditionally with an EFI / full disk image pair.
In addition, make error handling in qemu more robust, so failures are both
surfaced and do not cause endless attempts to allocate.
Please update image-garden to at least 5a00ead9964df6463e19432ae50e7760fc6da755
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The test adds a very small and simple smoke test that shows that a mount rule
with both fstype and options allows mounts to be performed on a real running
kernel.
The test is structured in a way that should make it easy to extend with new
variants (flags, fstype) in the future.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
- Tests defined in utils/test are now described by a task.yaml in the same
directory and can run concurrently across many machines.
- Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is
no longer a hard dependency in master.
- Tests no longer run on openSUSE Leap 15.6 due to the age of default
Python (3.6) and gcc/g++. The tight integration with SWIG which does
not seem to support other Python versions very well. Perl hard-codes
old GCC for extension modules. The upcoming openSUSE Leap 16 should be
a viable target. In the meantime we can still test everything through
rolling-release Tumbleweed.
- Formatting of YAML files is now more uniform, at four spaces per tab.
- The run-spread.sh script is now in the root of the tree. The script allows
running all spread tests sequentially on one system, while collecting logs
and artifacts for convenient analysis after the fact.
- All systems are adjusted to run _four_ workers in parallel with _two_ virtual
cores each and equipped with 1.5GB of virtual memory. This aims to best
utilize the capacity of a typical CI worker with two to four cores and about
8GB of available memory.
- Failing tests are marked as such, so that as a whole the entire spread suite
can pass and be useful at catching regressions.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Compared to v1 the following improvements have been made:
- The cost of installing packages have been shifted from each startup to image
preparation phase, thanks to the integration of custom cloud-init profiles
into image-garden. This has dramatic impact on iteration time while also
entirely removing requirement to be online to run once a prepared image is
available.
- Support for running on Google Compute Engine has been removed since it would
not be able to use cloud-init the same way would currently only complicate
setup.
- The number of workers have been tuned for local iteration, aiming for
comfortable work with 16GB of memory on the host. Once CI/CD pipeline
support is introduced I will add a dedicated entry so that resources are
utilized well both locally and when running in CI.
- The set of regression tests listed in tests/regression/apparmor/task.yaml is
now cross-checked so introduction of a new test to the makefile there is
automatically flagged and causes spread to fail with a clear message.
- The task tests/unit/utils has been improved to generate profiles. Thanks to
Christian Boltz for explaining this relationship between tests.
- A number of comments have been improved and cleaned up for readability,
accuracy and sometimes better grammar.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
