- Tests defined in utils/test are now described by a task.yaml in the same
directory and can run concurrently across many machines.
- Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is
no longer a hard dependency in master.
- Tests no longer run on openSUSE Leap 15.6 due to the age of default
Python (3.6) and gcc/g++. The tight integration with SWIG which does
not seem to support other Python versions very well. Perl hard-codes
old GCC for extension modules. The upcoming openSUSE Leap 16 should be
a viable target. In the meantime we can still test everything through
rolling-release Tumbleweed.
- Formatting of YAML files is now more uniform, at four spaces per tab.
- The run-spread.sh script is now in the root of the tree. The script allows
running all spread tests sequentially on one system, while collecting logs
and artifacts for convenient analysis after the fact.
- All systems are adjusted to run _four_ workers in parallel with _two_ virtual
cores each and equipped with 1.5GB of virtual memory. This aims to best
utilize the capacity of a typical CI worker with two to four cores and about
8GB of available memory.
- Failing tests are marked as such, so that as a whole the entire spread suite
can pass and be useful at catching regressions.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Compared to v1 the following improvements have been made:
- The cost of installing packages have been shifted from each startup to image
preparation phase, thanks to the integration of custom cloud-init profiles
into image-garden. This has dramatic impact on iteration time while also
entirely removing requirement to be online to run once a prepared image is
available.
- Support for running on Google Compute Engine has been removed since it would
not be able to use cloud-init the same way would currently only complicate
setup.
- The number of workers have been tuned for local iteration, aiming for
comfortable work with 16GB of memory on the host. Once CI/CD pipeline
support is introduced I will add a dedicated entry so that resources are
utilized well both locally and when running in CI.
- The set of regression tests listed in tests/regression/apparmor/task.yaml is
now cross-checked so introduction of a new test to the makefile there is
automatically flagged and causes spread to fail with a clear message.
- The task tests/unit/utils has been improved to generate profiles. Thanks to
Christian Boltz for explaining this relationship between tests.
- A number of comments have been improved and cleaned up for readability,
accuracy and sometimes better grammar.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
When the test name and test binary differed and genprofile was used, there would be an execname warning about the original expected binary not existing. This fixes that warning.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
The regression test suite uses root with capabilities restricted in
several tests. This can cause the test suite to fail in weird and
confusing ways.
Add a test to check for DAC permissiosns from / to the testsuite
and abort running the tests with an error message if DAC permissions
are going to cause the test suite to fail.
Currently the test is pretty basic, but is better than nothing.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1411
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: John Johansen <john@jjmx.net>
This fixes the test to pass on openSUSE Tumbleweed, where the small size
prevented alloction of an inode for the `lost+found` directory:
```
garden:opensuse-cloud-tumbleweed .../tests/regression/apparmor# mkfs.ext2 -F -m 0 -N 10 /tmp/sdtest.32929-21402-6x826m/image.ext3
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 512 1k blocks and 8 inodes
Allocating group tables: done
Writing inode tables: done
ext2fs_mkdir: Could not allocate inode in ext2 filesystem while creating /lost+found
```
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Which is technically not POSIX and command -v works everywhere. This fixes
building and running the test suite on openSUSE Tumbleweed.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The file was quoted with the following space, making the test broken.
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1429
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
Most `tests/regression/apparmor/*.sh` scripts contain
. $bin/prologue.inc
This will explode if one of the parent directories contains a space.
Minimized reproducer:
```
# cat test.sh
pwd=`dirname $0`
pwd=`cd $pwd ; /bin/pwd`
bin=$pwd
echo "pwd: $bin"
. $bin/prologue.inc
# ./test.sh
pwd: /tmp/foo bar
./test.sh: line 9: /tmp/foo: No such file or directory
```
Notice that test.sh tries to source `/tmp/foo` instead of `/tmp/foo bar/prologue.inc`.
The fix is to quote the prologue.inc path:
. "$bin/prologue.inc"
While on it, also fix other uses of $bin - directly and indirectly - by quoting them.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1418
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
The regression test suite uses root with capabilities restricted in
several tests. This can cause the test suite to fail in weird and
confusing ways.
Add a test to check for DAC permissiosns from / to the testsuite
and abort running the tests with an error message if DAC permissions
are going to cause the test suite to fail.
Currently the test is pretty basic, but is better than nothing.
Signed-off-by: John Johansen <john.johansen@canonical.com>
... to avoid issues with spaces in a parent directory's name.
"Indirect uses" means usage of $bin via another variable, for example
`foo=$bin/whatever`
Most `tests/regression/apparmor/*.sh` scripts contain
. $bin/prologue.inc
This will explode if one of the parent directories contains a space.
Minimized reproducer:
```
pwd=`dirname $0`
pwd=`cd $pwd ; /bin/pwd`
bin=$pwd
echo "pwd: $bin"
. $bin/prologue.inc
pwd: /tmp/foo bar
./test.sh: line 9: /tmp/foo: No such file or directory
```
Notice that test.sh tries to source `/tmp/foo` instead of `/tmp/foo bar/prologue.inc`.
The fix - as done in this commit - is to quote the prologue.inc path:
. "$bin/prologue.inc"
The only use of this _clone function passes in the same function that was
hardcoded, so this doesn't change any functionality.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
Even if file descriptor values would not exercise the full range provided
by int, it doesn't hurt to allocate enough space for all ints.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
If the test ran under a fs mounted with nosuid option, then these bits
would be ignored and the test would fail. In that case, detect it and
run the test in a tmpfs mountpoint without nosuid.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
When /tmp is of type tmpfs, the test didn't run because you can't
mount a swapfile on it. This patch mounts an ext2 mountpoint on
$tmpdir so that the swapfile can be mounted on top of it instead of
tmpfs.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
When /tmp is mounted, remounting / as private for tests that don't
work when shared still fail because /tmp remains as shared. The option
-T in findmnt helps determine the mountpoint in a certain directory,
so use that with $tmpdir to determine the root.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
The tests that use pivot_root or move mountpoints with mount have to
make sure that / is private for the tests to work. Refactor that logic
into a file to be sourced by the test scripts
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Without AA_MAY_MOUNT, mount was not allowed by the allow all
rule. AA_DUMMY_REMOUNT does become AA_MAY_MOUNT, but it fixes the
flags to remount only, so other options are not included. Also, add
allow all rule testcases to the mount regression tests.
Fixes: https://gitlab.com/apparmor/apparmor/-/issues/410
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
By specifying 0 in the unix type, all rules were allowing only the
"none" type, when it wanted to allow all types, so replace it by
0xffffffff. Also, add this testcase to the unix regression tests.
Fixes: https://gitlab.com/apparmor/apparmor/-/issues/410
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Add infrastructure for calling the mount test binary with an fstype
instead of using the default hardcoded ext2 type, and then use that in a
test that exercises CVE-2016-1585, ensuring that mounting a procfs
filesystem isn't permitted when the only mount rule is
mount options=(rw,make-slave) -> **,
to try to ensure that the generated and enforced policy is restricted to
what is intended.
Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Bug: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1597017
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1211
When the setup of the notify options failed, they were exiting the
program without cleaning up the mqueue. Fix this by returning instead
of exiting, since the main function does the cleanup in case of any
failures. If the test succeeds, then it exits successfully.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
mq_notify only notifies if the queue is empty, so if the sender wins
the race and sends a message before mq_notify is set up, mq_notify
will timeout.
Adding synchronization using pipes the same way it was used in the
setns tests should fix it. The pipe now needs rw permissions, so add
that to the tests.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
The child which sends the message was winning the race and causing a
timeout when the receiver was waiting for the message.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
net_inet makes more sense since other finegrained network types can be
added in the future.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
By not having quotes in $@, the string splits by the whitespace.
That prevents us from checking if the parser supports rules
that have spaces in them.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
There is a kernel feature, available under
namespaces/userns_create/pciu&, that enables the transition of
unconfined tasks to a special profile called unprivileged_userns when
they try to create an unprivileged user namespace with
clone/unshare. This transition allows the creation of the unprivileged
user namespace but hinders its privileges by not allowing
capabilities. Refer to the unprivileged_userns profile to check what
rules are allowed.
If either the feature is not present in the kernel, or the
unprivileged_userns profile is not loaded, then the defined behavior
is to deny the creation of the unprivileged user namespace
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Some of the tests are failing because of /usr/bin/true vs /bin/true.
Similarly to what was done in 8c09b328, to make the tests more
reliable, copy the true binary to $tmpdir and use this path on the
tests instead.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
There are some tests like attach_disconnected and posix_mq that can
have a program that calls another. For example, posix_mq_rcv calls
posix_mq_snd. Both of them write to the same output file, but the code
that checks the result expects only one line. This change enables
checking multiple lines in the output file and passing or failing
accordingly.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1140
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
There are some tests like attach_disconnected and posix_mq that can
have a program that calls another. For example, posix_mq_rcv calls
posix_mq_snd. Both of them write to the same output file, but the code
that checks the result expects only one line. This change enables
checking multiple lines in the output file and passing or failing
accordingly.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
The move_mount tests were returning -1 in case of failure causing it
to become 255 in some systems, but checktestbg in the testsuite
considers any return value greater than 128 to be a signal error.
That would cause tests that should fail to display the following test
error:
... was expected to 'fail'. Reason for failure 'killed by
signal 127'
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
the stacking tests need to be able to read and write the new apparmor
dir in proc, if that interface has been selected. Update the tests to
make sure they have the permissions needed.
Signed-off-by: John Johansen <john.johansen@canonical.com>
upstream kernels only have network_v8 unfortunately the tcp tests were
only being run against kernels that had network (which is v7). Kernels
that support both (Ubuntu) would be tested against v8, so v8 has been
tested but pure upstream kernels were failing to be tested correctly.
This patch will only make sure one of the supported versions are
tested. This is determined by the parser which prefers v8. In the
future the tests need to be extended to run the tests against all
kernel supported versions.
Signed-off-by: John Johansen <john.johansen@canonical.com>
