- Tests defined in utils/test are now described by a task.yaml in the same
directory and can run concurrently across many machines.
- Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is
no longer a hard dependency in master.
- Tests no longer run on openSUSE Leap 15.6 due to the age of default
Python (3.6) and gcc/g++. The tight integration with SWIG which does
not seem to support other Python versions very well. Perl hard-codes
old GCC for extension modules. The upcoming openSUSE Leap 16 should be
a viable target. In the meantime we can still test everything through
rolling-release Tumbleweed.
- Formatting of YAML files is now more uniform, at four spaces per tab.
- The run-spread.sh script is now in the root of the tree. The script allows
running all spread tests sequentially on one system, while collecting logs
and artifacts for convenient analysis after the fact.
- All systems are adjusted to run _four_ workers in parallel with _two_ virtual
cores each and equipped with 1.5GB of virtual memory. This aims to best
utilize the capacity of a typical CI worker with two to four cores and about
8GB of available memory.
- Failing tests are marked as such, so that as a whole the entire spread suite
can pass and be useful at catching regressions.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Compared to v1 the following improvements have been made:
- The cost of installing packages have been shifted from each startup to image
preparation phase, thanks to the integration of custom cloud-init profiles
into image-garden. This has dramatic impact on iteration time while also
entirely removing requirement to be online to run once a prepared image is
available.
- Support for running on Google Compute Engine has been removed since it would
not be able to use cloud-init the same way would currently only complicate
setup.
- The number of workers have been tuned for local iteration, aiming for
comfortable work with 16GB of memory on the host. Once CI/CD pipeline
support is introduced I will add a dedicated entry so that resources are
utilized well both locally and when running in CI.
- The set of regression tests listed in tests/regression/apparmor/task.yaml is
now cross-checked so introduction of a new test to the makefile there is
automatically flagged and causes spread to fail with a clear message.
- The task tests/unit/utils has been improved to generate profiles. Thanks to
Christian Boltz for explaining this relationship between tests.
- A number of comments have been improved and cleaned up for readability,
accuracy and sometimes better grammar.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Remove conditional logic from the parser and move it to its own class,
that way any improvements or conditional features will make cleaner
changes.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Imported from https://github.com/alexmurray/apparmor-mode with just a minimal
change to update the URL field to point to the upstream apparmor repo.
Signed-off-by: Alex Murray <alex.murray@canonical.com>
Add a flag that allows setting the error code AppArmor will send when
an operation is denied. This should not be used normally.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
net_inet makes more sense since other finegrained network types can be
added in the future.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
Extend the policy syntax to have a rule that allows specifying all
permissions for all rule types.
allow all,
This is useful for making blacklist based policy, but can also be
useful when combined with other rule prefixes, eg. to add audit
to all rules.
audit access all,
Signed-off-by: John Johansen <john.johansen@canonical.com>
aa-load is a tool that loads cached (compiled) policies into
the kernel. It can receive as argument a file, a cache directory
containing the hash subtree, and a directory containing cached
files directly underneath - no hash.
This tool can be used in the as a guide for other init
systems to load the cached policies directly.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/770
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
Setns is used to associate to an existing user namespace, so the
kernel security hook for user namespace creation is not called.
The restriction for setns is that it should have the capability
sys_admin.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
This test uses unix_fd_server to open a file and pass
its file descriptor to the attach_disconnected tests, which
then mounts, pivots root and then tries to open the file.
Since the server execs the client, this commit also inverts the order
of the parameters to allow the server to forward the client's args
along with the unix_socket path.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/810
Acked-by: John Johansen <john.johansen@canonical.com>
The code from unix_fd_client can be used by other clients which
will receive the fd of a file using SCM rights.
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/810
Acked-by: John Johansen <john.johansen@canonical.com>
The Hack used to build the libapparmor swig interface for ruby fails
with ruby 3.1. Instead of trying to do black magic in ruby to rename
the generated Makefile to Makefile.new, just save off the Makefile
and restore after ruby's setup has been called.
Fixes: https://gitlab.com/apparmor/apparmor/-/issues/206
Signed-off-by: John Johansen <john.johansen@canonical.com>
Acked-by: Christian Boltz <apparmor@cboltz.de>
aa-load is a tool that loads cached (compiled) policies into
the kernel. It can receive as argument a file, a cache directory
containing the hash subtree, and a directory containing cached
files directly underneath - no hash.
This tool can be used in the as a guide for other init
systems to load the cached policies directly.
Signed-off-by: John Johansen <john.johansen@canonical.com>
Signed-off-by: Georgia Garcia <georgia.garcia@canonical.com>
The auto-generated cap_names.h has problems when the parser if the
parser is built against a kernel with a smaller capability list than
the kernel policy is being compiled for.
Moving to a pre-generated list lets us support all capabilities even
when we build against older kernels. However we don't want to only use
the pre-generated list as that would make it too easy to miss when a
new capability has been added.
Keep auto generating the caps list and compare it to the pre-generated
caps list so we can detect when new capabilities are added, and fail
the build so that the pre-generated list can be updated. We screen the
diff for only additions so that the parser can continue to build on
older kernels that don't have the full capability list without errors.
Signed-off-by: John Johansen <john.johansen@canonical.com>
The old patter *.* doesn't match lsb_release and nvidia_modprobe, and
the only file we ship in local is a README. This patch adjusts the
pattern to ignore everything except README.
Ignoring cscope.* files allows users of cscope to not be bothered by
`git status` reporting that an unknown file is in the source tree.
Signed-off-by: Tyler Hicks <tyhicks@canonical.com>
