Allow running tests with spread
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
|
|
|
project: apparmor
|
|
|
|
backends:
|
2024-11-26 15:40:19 +01:00
|
|
|
garden:
|
|
|
|
# The garden backend relies on https://gitlab.com/zygoon/image-garden
|
|
|
|
# TODO: Switch to a released version for better stability.
|
|
|
|
type: adhoc
|
|
|
|
# Use just enough RAM to link the parser on a virtual system with two
|
|
|
|
# cores. Using more cores may easily consume more memory, due to make
|
|
|
|
# -j$(nproc, used below than a small CI/CD system is typically granted.
|
|
|
|
# It is better to have more workers than to have one big worker with
|
|
|
|
# lots of resources.
|
|
|
|
allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 1536" QEMU_SMP_OPTION="-smp 2" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))"
|
|
|
|
discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS"
|
|
|
|
systems:
|
|
|
|
# All systems except for the one Ubuntu system are marked as
|
|
|
|
# manual. This way we don't accidentally spin up everything when
|
|
|
|
# someone runs spread without knowing better.
|
|
|
|
- opensuse-cloud-tumbleweed:
|
|
|
|
username: opensuse
|
|
|
|
password: opensuse
|
|
|
|
workers: 4
|
|
|
|
manual: true
|
|
|
|
- debian-cloud-12:
|
|
|
|
username: debian
|
|
|
|
password: debian
|
|
|
|
workers: 4
|
|
|
|
manual: true
|
|
|
|
- debian-cloud-13:
|
|
|
|
username: debian
|
|
|
|
password: debian
|
|
|
|
workers: 4
|
|
|
|
manual: true
|
|
|
|
- ubuntu-cloud-22.04:
|
|
|
|
username: ubuntu
|
|
|
|
password: ubuntu
|
|
|
|
workers: 4
|
|
|
|
manual: true
|
|
|
|
- ubuntu-cloud-24.04:
|
|
|
|
username: ubuntu
|
|
|
|
password: ubuntu
|
|
|
|
workers: 4
|
|
|
|
manual: true
|
|
|
|
- ubuntu-cloud-24.10:
|
|
|
|
username: ubuntu
|
|
|
|
password: ubuntu
|
|
|
|
workers: 4
|
Allow running tests with spread
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
|
|
|
|
|
|
|
exclude:
|
2024-11-26 15:40:19 +01:00
|
|
|
- .git
|
|
|
|
- "*.o"
|
|
|
|
# Files related to spread and image-garden.
|
|
|
|
- "*.qcow2"
|
|
|
|
- "*.iso"
|
|
|
|
- "*.img"
|
|
|
|
- "*.log"
|
|
|
|
- "*.run"
|
|
|
|
- "*.lock"
|
|
|
|
- spread-logs
|
|
|
|
- spread-artifacts
|
Allow running tests with spread
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
|
|
|
|
|
|
|
# Copy the project to this path on the test system.
|
|
|
|
# This is also available as $SPREAD_PATH.
|
|
|
|
path: /tmp/apparmor
|
|
|
|
|
|
|
|
prepare: |
|
2024-11-26 15:40:19 +01:00
|
|
|
# Configure libapparmor but only if a makefile is not already present.
|
|
|
|
# This makes repeated iteration with -reuse much faster, as the chain of
|
|
|
|
# invocations of make below are efficient if nothing needs to be done.
|
|
|
|
if [ ! -f "$SPREAD_PATH"/libraries/libapparmor/Makefile ]; then
|
|
|
|
(
|
|
|
|
cd $SPREAD_PATH/libraries/libapparmor
|
|
|
|
sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python
|
|
|
|
)
|
|
|
|
fi
|
|
|
|
# Build libapparmor.
|
|
|
|
make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)"
|
|
|
|
# Build apparmor_parser.
|
|
|
|
make -C $SPREAD_PATH/parser -j"$(nproc)"
|
|
|
|
# Build binary utilities (aa-exec and firends).
|
|
|
|
make -C $SPREAD_PATH/binutils -j"$(nproc)"
|
|
|
|
# Build python utilities.
|
|
|
|
make -C $SPREAD_PATH/utils -j"$(nproc)"
|
Allow running tests with spread
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
|
|
|
|
|
|
|
# In case of failure, include the kernel version in the log.
|
|
|
|
debug-each: |
|
2024-11-26 15:40:19 +01:00
|
|
|
uname -a
|
Allow running tests with spread
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
2024-11-25 13:54:03 +01:00
|
|
|
|
|
|
|
suites:
|
2024-11-26 15:40:19 +01:00
|
|
|
utils/:
|
|
|
|
summary: Unit tests for the Python utilities.
|
|
|
|
prepare: |
|
|
|
|
# Generate apparmor profiles that the tests rely on.
|
|
|
|
make -C "$SPREAD_PATH"/parser/tst gen_xtrans gen_dbus
|
|
|
|
# Spread does not support programmatically generated test variants.
|
|
|
|
# Ensure that the list baked into utils/test/task.yaml contains all
|
|
|
|
# the files matching utils/test/test-*.py
|
|
|
|
fail=0
|
|
|
|
for V in $SPREAD_PATH/utils/test/test-*.py; do
|
|
|
|
Vdash="$(basename "$V" | sed -e 's,^test-,,' -e 's,\.py$,,')"
|
|
|
|
Vunder="$(basename "$V" | sed -e 's,^test-,,' -e 's,\.py$,,' -e 's,-,_,g')"
|
|
|
|
if ! grep -xF ' TEST/'"$Vunder"': '"$Vdash" "$SPREAD_PATH"/utils/test/task.yaml; then
|
|
|
|
echo "utils/test/task.yaml: missing test variant: TEST/$Vunder: $Vdash" >&2
|
|
|
|
fail=1
|
|
|
|
fi
|
|
|
|
done
|
|
|
|
if [ "$fail" -ne 0 ]; then
|
|
|
|
echo "exiting due to missing variants listed above" >&2
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
tests/unit/:
|
|
|
|
summary: Unit tests that do not exercise the kernel layer.
|
|
|
|
tests/regression/:
|
|
|
|
summary: Regression tests for parser-kernel interaction.
|
|
|
|
prepare: |
|
|
|
|
# Spread does not support programmatically generated test variants.
|
|
|
|
# Ensure that the list baked into tests/regression/apparmor/task.yaml
|
|
|
|
# contains all the tests defined in tests/regression/apparmor/Makefile.
|
|
|
|
echo '$(foreach t,$(TESTS),$(info TEST/$t))' | \
|
|
|
|
make -n -f "$SPREAD_PATH"/tests/regression/apparmor/Makefile -f /dev/stdin | \
|
|
|
|
grep -F TEST/ | \
|
|
|
|
cut -d / -f 2 | \
|
|
|
|
tee apparmor-regression-tests.txt
|
|
|
|
fail=0
|
|
|
|
while read -r V; do
|
|
|
|
if ! grep -xF ' TEST/'"$V"': 1' "$SPREAD_PATH"/tests/regression/apparmor/task.yaml; then
|
|
|
|
echo "tests/regression/task.yaml: missing test variant: TEST/$V" >&2
|
|
|
|
fail=1
|
|
|
|
fi
|
|
|
|
done <apparmor-regression-tests.txt
|
|
|
|
if [ "$fail" -ne 0 ]; then
|
|
|
|
echo "exiting due to missing variants listed above" >&2
|
|
|
|
exit 1
|
|
|
|
fi
|
2024-11-26 01:15:51 +01:00
|
|
|
|
2024-11-26 15:40:19 +01:00
|
|
|
# Build all the apparmor regression test programs.
|
|
|
|
make -C "$SPREAD_PATH"/tests/regression/apparmor -j"$(nproc)"
|
|
|
|
restore: |
|
|
|
|
rm -f apparmor-regression-tests.txt
|