The test adds a very small and simple smoke test that shows that a mount rule
with both fstype and options allows mounts to be performed on a real running
kernel.
The test is structured in a way that should make it easy to extend with new
variants (flags, fstype) in the future.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1445
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
When settest was called with two parameters, one for the test name and
the other for the test wrapper/binary, the profile created with
genprofile would show the test name, causing an error if the file
didn't exist.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Some of the tests using the --stdin option of mkprofile.pl are adding
more than one profile at a time. Whenever a profile is created in the
test, its name is added to the file profile.names so the test
infrastructure can tell if the profile is loaded or removed when
appropriately. The issue is that the name of the second profile
created by --stdin is not added, so these checks are not applied.
This patch adds the option of appending a second profile (not rules).
The option --append was used instead of a short -A because the short
options are arguments of mkprofile.pl, which --append is not.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
The test adds a very small and simple smoke test that shows that a mount rule
with both fstype and options allows mounts to be performed on a real running
kernel.
The test is structured in a way that should make it easy to extend with new
variants (flags, fstype) in the future.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
- previously, aa-status --json --show profiles would return non-standard json
- adding the --pretty flag would crash completely
- closes#470
Things done:
- removed trailing ", " in json generation
- generate json seperator (", ") for each new json field
(profiles/processes) after the header if json is enabled
Tested on NixOS and apparmor 4.0.3 base, but should work on any version the patch applies on.
Due to how the tests were implemented in the past, permissions could
be passed along with the image name, and the permission part would be
discarded. The issue is that permissions are usually separated by ':',
but namespaces also contain ':', which would cause a conflict.
Since permissions are no longer passed as part of the image name,
remove that description so profile names in namespaces can be
supported.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1432
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
It's no longer in python standard library starting
at version 3.13. Fixes:
root@qemuarm64:~# aa-complain /etc/apparmor.d/*
Traceback (most recent call last):
File "/usr/sbin/aa-complain", line 18, in <module>
from apparmor.fail import enable_aa_exception_handler
File "/usr/lib/python3.13/site-packages/apparmor/fail.py", line 12, in <module>
import cgitb
ModuleNotFoundError: No module named 'cgitb'
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Tests #466 but is marked as expected fail due to that bug not being resolved.
Depends on !1441 which adds the xfail infrastructure to the parser equality testing framework, and should be rebased on top of master once that MR is merged.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1443
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
- Tests defined in utils/test are now described by a task.yaml in the same
directory and can run concurrently across many machines.
- Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is
no longer a hard dependency in master.
- Tests no longer run on openSUSE Leap 15.6 due to the age of default
Python (3.6) and gcc/g++. The tight integration with SWIG which does
not seem to support other Python versions very well. Perl hard-codes
old GCC for extension modules. The upcoming openSUSE Leap 16 should be
a viable target. In the meantime we can still test everything through
rolling-release Tumbleweed.
- Formatting of YAML files is now more uniform, at four spaces per tab.
- The run-spread.sh script is now in the root of the tree. The script allows
running all spread tests sequentially on one system, while collecting logs
and artifacts for convenient analysis after the fact.
- All systems are adjusted to run _four_ workers in parallel with _two_ virtual
cores each and equipped with 1.5GB of virtual memory. This aims to best
utilize the capacity of a typical CI worker with two to four cores and about
8GB of available memory.
- Failing tests are marked as such, so that as a whole the entire spread suite
can pass and be useful at catching regressions.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Compared to v1 the following improvements have been made:
- The cost of installing packages have been shifted from each startup to image
preparation phase, thanks to the integration of custom cloud-init profiles
into image-garden. This has dramatic impact on iteration time while also
entirely removing requirement to be online to run once a prepared image is
available.
- Support for running on Google Compute Engine has been removed since it would
not be able to use cloud-init the same way would currently only complicate
setup.
- The number of workers have been tuned for local iteration, aiming for
comfortable work with 16GB of memory on the host. Once CI/CD pipeline
support is introduced I will add a dedicated entry so that resources are
utilized well both locally and when running in CI.
- The set of regression tests listed in tests/regression/apparmor/task.yaml is
now cross-checked so introduction of a new test to the makefile there is
automatically flagged and causes spread to fail with a clear message.
- The task tests/unit/utils has been improved to generate profiles. Thanks to
Christian Boltz for explaining this relationship between tests.
- A number of comments have been improved and cleaned up for readability,
accuracy and sometimes better grammar.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The new check-one-test-% pattern rule allows running individual test scripts.
This allows them to be tested in parallel across many Make worker threads or
across many distinct machines with spread.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This reverts merge request !1446 due to breakage in the aa-exec and userns regression tests.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1447
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
When the test name and test binary differed and genprofile was used, there would be an execname warning about the original expected binary not existing. This fixes that warning.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1446
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
When the test name and test binary differed and genprofile was used, there would be an execname warning about the original expected binary not existing. This fixes that warning.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
currently the equality tests require the tests to PASS as known equality
or inequality. Add the ability to add tests that are a known problem
and are expected to fail the equality, or inequality test.
This is done by using
verify_binary_xequality
verify_binary_xinequality
This allows new tests to be added to document a known issue, without
having to develop the fix for the issue. The use of this facility
is expected to be temporary, so any test marked as xequality or
xinequality will be noisy but not fail the other tests until they
are fixed, at which point they will cause the tests to fail to
force them to be updated to the correct equality or inequality
test.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1441
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
Update the bwrap profile so that it will attach to application profiles
if present.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1435
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: John Johansen <john@jjmx.net>
currently the equality tests require the tests to PASS as known equality
or inequality. Add the ability to add tests that are a known problem
and are expected to fail the equality, or inequality test.
This is done by using
verify_binary_xequality
verify_binary_xinequality
This allows new tests to be added to document a known issue, without
having to develop the fix for the issue. The use of this facility
is expected to be temporary, so any test marked as xequality or
xinequality will be noisy but not fail the other tests until they
are fixed, at which point they will cause the tests to fail to
force them to be updated to the correct equality or inequality
test.
Signed-off-by: John Johansen <john.johansen@canonical.com>
No need to assign a variable to itsself, not even conditionally.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1442
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
Somehow the use of new match statements slipped by review despite our commitment to supporting older Python versions. Replace them with an unfortunately-needed if-elif chain.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1440
Approved-by: Christian Boltz <apparmor@cboltz.de>
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
I am upstreaming this patch that is part of the nix package of apparmor for close to a year now.
This fixes the issue at https://github.com/NixOS/nixpkgs/issues/273164 for more distros than just NixOS.
The original merge Request on the nix side patching this was https://github.com/NixOS/nixpkgs/pull/285915.
However, people had issues with gitlab, so this never hit apparmor upstream until now. This does however also mean this patch has seen production and seems to work quite well.
## Original reasoning/message of the patch author:
This check is intended for ensuring that the profiles file can actually
be opened. The *actual* check is performed by the shell, not the read
utility, which won't even be executed if the input redirection (and
hence the test) fails.
If the test succeeds, though, using `read` here might actually
jeopardize the test result if there are no profiles loaded and the file
is empty.
This commit fixes that case by simply using `true` instead of `read`.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1438
Approved-by: Christian Boltz <apparmor@cboltz.de>
Merged-by: Christian Boltz <apparmor@cboltz.de>
This check is intended for ensuring that the profiles file can actually
be opened. The *actual* check is performed by the shell, not the read
utility, which won't even be executed if the input redirection (and
hence the test) fails.
If the test succeeds, though, using `read` here might actually
jeopardize the test result if there are no profiles loaded and the file
is empty.
This commit fixes that case by simply using `true` instead of `read`.
As pointed out by https://bugs.launchpad.net/apparmor/+bug/2087875 ,
profile transitions with pivot_root are currently not supported on any
kernel.
This commit makes this limitation more obvious to users.
Signed-off-by: Maxime Bélair <maxime.belair@canonical.com>
This MR documents the lessons learned from the experiments that ultimately resulted in !1416.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1434
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
This also includes renaming SIGTSTP "stp" to "tstp" while preserving backwards compatibility.
Analogous to !1420.
Signed-off-by: Ryan Lee <ryan.lee@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1425
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: John Johansen <john@jjmx.net>
The regression test suite uses root with capabilities restricted in
several tests. This can cause the test suite to fail in weird and
confusing ways.
Add a test to check for DAC permissiosns from / to the testsuite
and abort running the tests with an error message if DAC permissions
are going to cause the test suite to fail.
Currently the test is pretty basic, but is better than nothing.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1411
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: John Johansen <john@jjmx.net>
The new flag --merge-notifications enables the merging of all
notifications from a fixed time period into a single one, thus
preventing notification flooding.
A new GUI allows users to choose either a synthetic or a comprehensive
view of the notifications.
Signed-off-by: Maxime Bélair <maxime.belair@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1324
Approved-by: Christian Boltz <apparmor@cboltz.de>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
I've been working on improved end-to-end testing of AppArmor on a number
of popular Linux distributions. My first run contains Debian, Ubuntu and openSUSE.
This branch contains three small fixes that, mainly, allow running more tests on
openSUSE Tumbleweed.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1431
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Approved-by: Christian Boltz <apparmor@cboltz.de>
Merged-by: Christian Boltz <apparmor@cboltz.de>
Using this version of make:
```
GNU Make 4.2.1
Built for x86_64-suse-linux-gnu
```
I'm not entirely sure why but the alternative syntax I've used works correctly.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This fixes the test to pass on openSUSE Tumbleweed, where the small size
prevented alloction of an inode for the `lost+found` directory:
```
garden:opensuse-cloud-tumbleweed .../tests/regression/apparmor# mkfs.ext2 -F -m 0 -N 10 /tmp/sdtest.32929-21402-6x826m/image.ext3
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 512 1k blocks and 8 inodes
Allocating group tables: done
Writing inode tables: done
ext2fs_mkdir: Could not allocate inode in ext2 filesystem while creating /lost+found
```
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This fixes an error with Python 3.11:
```
test/test-parser-simple-tests.py:420:21: E502 the backslash is redundant between brackets
```
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Which is technically not POSIX and command -v works everywhere. This fixes
building and running the test suite on openSUSE Tumbleweed.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
On a test system without bison installed, make setup fails with:
/bin/sh: 1: bison: not found
/bin/sh: 1: test: -ge: unexpected operator
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The file was quoted with the following space, making the test broken.
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1429
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: Georgia Garcia <georgia.garcia@canonical.com>
