apparmor/spread.yaml

123 lines
4.1 KiB
YAML
Raw Normal View History

Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
project: apparmor
backends:
garden:
# The garden backend relies on https://gitlab.com/zygoon/image-garden
# TODO: Switch to a released version for better stability.
type: adhoc
# Use 2GB of RAM and four cores as otherwise we may not have enough memory
# to link the parser. It is better to have more workers than to have one
# big worker with lots of resources.
allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 4" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))"
discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS"
systems:
# All systems except for the one Ubuntu system are marked as manual.
# This way we don't accidentally spin up everything when someone runs
# spread without knowing better.
- opensuse-cloud-15.6:
username: opensuse
password: opensuse
workers: 2
manual: true # Run only when explicitly named.
environment:
# openSUSE 15 ships very old default python.
PYTHON: /usr/bin/python3.11
PYTHON_CONFIG: /usr/bin/python3.11-config
- opensuse-cloud-tumbleweed:
username: opensuse
password: opensuse
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
manual: true
- debian-cloud-12:
username: debian
password: debian
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
manual: true
- debian-cloud-13:
username: debian
password: debian
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
manual: true
- ubuntu-cloud-22.04:
username: ubuntu
password: ubuntu
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
manual: true
- ubuntu-cloud-24.04:
username: ubuntu
password: ubuntu
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
manual: true
- ubuntu-cloud-24.10:
username: ubuntu
password: ubuntu
workers: 4
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
exclude:
- .git
- "*.o"
# Files related to spread and image-garden.
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
- "*.qcow2"
- "*.iso"
- "*.img"
- "*.log"
- "*.run"
- "*.lock"
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
# Copy the project to this path on the test system.
# This is also available as $SPREAD_PATH.
path: /tmp/apparmor
prepare: |
# Configure libapparmor but only if a makefile is not already present.
# This makes repeated iteration with -reuse much faster, as the chain of
# invocations of make below are efficient if nothing needs to be done.
if [ ! -f "$SPREAD_PATH"/libraries/libapparmor/Makefile ]; then
(
cd $SPREAD_PATH/libraries/libapparmor
sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python
)
fi
Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
# Build libapparmor.
make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)"
# Build apparmor_parser.
make -C $SPREAD_PATH/parser -j"$(nproc)"
# Build binary utilities (aa-exec and firends).
make -C $SPREAD_PATH/binutils -j"$(nproc)"
# Build python utilities.
make -C $SPREAD_PATH/utils -j"$(nproc)"
# In case of failure, include the kernel version in the log.
debug-each: |
uname -a
suites:
tests/unit/:
summary: Unit tests that do not exercise the kernel layer.
tests/regression/:
summary: Regression tests for parser-kernel interaction.
prepare: |
# Spread does not support programmatically generated test variants.
# Ensure that the list baked into tests/regressin/apparmor/task.yaml
# contains all the tests defined in tests/regression/apparmor/Makefile.
echo '$(foreach t,$(TESTS),$(info TEST/$t))' | \
make -n -f "$SPREAD_PATH"/tests/regression/apparmor/Makefile -f /dev/stdin | \
grep -F TEST/ | \
cut -d / -f 2 | \
tee apparmor-regression-tests.txt
fail=0
for V in $(cat apparmor-regression-tests.txt); do
if ! grep -xF ' TEST/'"$V"': 1' "$SPREAD_PATH"/tests/regression/apparmor/task.yaml; then
echo "tests/regression/task.yaml: missing test variant: TEST/$V" >&2
fail=1
fi
done
if [ "$fail" -ne 0 ]; then
echo "exiting due to missing variants listed above" >&2
exit 1
fi
# Build all the apparmor regression test programs.
make -C "$SPREAD_PATH"/tests/regression/apparmor -j"$(nproc)"
restore: |
rm -f apparmor-regression-tests.txt