3 commits

Author	SHA1	Message	Date
Zygmunt Krynicki	1df91e2c8c	Third iteration of spread support ... - Tests defined in utils/test are now described by a task.yaml in the same directory and can run concurrently across many machines. - Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is no longer a hard dependency in master. - Tests no longer run on openSUSE Leap 15.6 due to the age of default Python (3.6) and gcc/g++. The tight integration with SWIG which does not seem to support other Python versions very well. Perl hard-codes old GCC for extension modules. The upcoming openSUSE Leap 16 should be a viable target. In the meantime we can still test everything through rolling-release Tumbleweed. - Formatting of YAML files is now more uniform, at four spaces per tab. - The run-spread.sh script is now in the root of the tree. The script allows running all spread tests sequentially on one system, while collecting logs and artifacts for convenient analysis after the fact. - All systems are adjusted to run _four_ workers in parallel with _two_ virtual cores each and equipped with 1.5GB of virtual memory. This aims to best utilize the capacity of a typical CI worker with two to four cores and about 8GB of available memory. - Failing tests are marked as such, so that as a whole the entire spread suite can pass and be useful at catching regressions. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>	2024-12-05 02:17:07 +01:00
Zygmunt Krynicki	c95ac4d350	Second iteration of spread support ... Compared to v1 the following improvements have been made: - The cost of installing packages have been shifted from each startup to image preparation phase, thanks to the integration of custom cloud-init profiles into image-garden. This has dramatic impact on iteration time while also entirely removing requirement to be online to run once a prepared image is available. - Support for running on Google Compute Engine has been removed since it would not be able to use cloud-init the same way would currently only complicate setup. - The number of workers have been tuned for local iteration, aiming for comfortable work with 16GB of memory on the host. Once CI/CD pipeline support is introduced I will add a dedicated entry so that resources are utilized well both locally and when running in CI. - The set of regression tests listed in tests/regression/apparmor/task.yaml is now cross-checked so introduction of a new test to the makefile there is automatically flagged and causes spread to fail with a clear message. - The task tests/unit/utils has been improved to generate profiles. Thanks to Christian Boltz for explaining this relationship between tests. - A number of comments have been improved and cleaned up for readability, accuracy and sometimes better grammar. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>	2024-12-05 02:17:07 +01:00
Zygmunt Krynicki	cc04181578	Allow running tests with spread ... Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>	2024-12-05 02:17:07 +01:00