It's too difficult to keep the podman-machine image up-to-date.
And, we can't use the cache on Mac/Windows, so if quay is down
we're hosed no matter what.
Add a "nocache" mechanism to install_test_configs() and use that
in machine test setup.
Signed-off-by: Ed Santiago <santiago@redhat.com>
As of https://github.com/containers/automation_images/pull/357
our CI VMs include a local registry preloaded with all(*)
images used in tests.
* where "all" means "most".
This commit installs a new registries.conf that redirects docker
and quay to the new local registry. The hope is that this will
reduce CI flakes.
Since tests change over time, and new tests may require new
images, this commit also adds a mechanism for pulling in
remote images at test run time. Obviously this negates
the purpose of the cache, since it introduces a flake
pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY
NECESSARY, and then, if we have to do this, hurry up and
spin new CI VMs that include the new image(s).
Signed-off-by: Ed Santiago <santiago@redhat.com>
Run root e2e & system tests using composefs on rawhide.
Write magic settings to storage.conf. That part is easy.
e2e tests, however, ignore storage.conf. They require everything
to be specified on the command line. And "everything", in the
case of composefs, includes a long complicated --pull-options
string which in turn requires containers-storage PR 1966
which, as of this writing, is finally vendored into podman.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Try to speed up the CI tests by using tmpfs as container storage.
This is important for system tests, other tests setup their own --root
already on tmpfs so it should not effect them.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Two enormous misunderstandings:
1) $CIRRUS_BASE_SHA is worthless. I thought it was, you know,
the BASE SHA of the current commit, but (as best I can tell)
it seems to be the SHA of the most recent commit on the
destination branch. Cirrus docs are unhelpful. Anyhow,
it's clearly not anything useful. Stop using it.
2) $EPOCH_TEST_COMMIT is closer to what we want. It is
defined in Makefile as the git merge-base. But for unknown
reasons it was being clobbered in CI scripts, and it
doesn't seem to work in all contexts, so, eliminate it
from CI setup scripts. Leave it only in Makefile.
This leaves us with no option other than defining our own
merge-base variable, PR_BASE_SHA. Do so and pass it along
to rootless jobs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
With (esp. Debian) CI VM images built by
https://github.com/containers/automation_images/ pull/338 CI no-longer
tests with runc nor cgroups v1. Add logic to fail under these
conditions. Prune back high-level YAML/script envars and logic formerly
required to support these things.
Signed-off-by: Chris Evich <cevich@redhat.com>
The leak check is slower (over 5mins) so we do not wnat them on PR runs
to speed system tests up. However that opens the door for someone to add
a test which forgets to do the correct cleanup themselves. This might
not cause a fatal error right away and only later when new tests would
be added. To prevent this happening the nighlty run will check leaks so
that we can fix them quickly and not notice them months/years later when
a new test is added that might trip over it.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
A long time ago, `passthrough_envars()` was defined in `lib.sh`. It has
since been moved, but the related comments were never updated. Update
the env. var. comments pointing future maintainers to the function that
relies on them. Otherwise a simple search w/in this repo. won't turn up
anything.
Signed-off-by: Chris Evich <cevich@redhat.com>
Podman v5 will not support cgroups-v1. This commit will print a warning
if it detects a cgroups-v1 system. The warning can be hidden by setting
envvar `PODMAN_CGROUPSV1_WARNING`.
This warning is patched out for RHEL 9 builds as cgroups-v1 will still
be supported on RHEL 9 systems.
Resolves: https://issues.redhat.com/browse/RUN-1957
[NO NEW TESTS NEEDED]
Co-authored-by: Ed Santiago <santiago@redhat.com>
Co-authored-by: Sascha Grunert <sgrunert@redhat.com>
Co-authored-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@redhat.com>
CNI is deprecated and is build tagged out for 5.0. Don't test it in our CI.
This commit also disables upgrade tests for now - those need more work since the old version of Podman only uses CNI. Upgrade tests will be re-vamped in a later commit.
Signed-off-by: Ashley Cui <acui@redhat.com>
We're only testing vfs in CI. That's bad. #18822 tried to
remedy that but that only worked on system tests, not e2e.
Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml
in the same vein as all the other CI_DESIRED_X. Since it's 2023
we default to overlay, testing vfs only in priorfedora.
Fixes required:
- e2e tests:
- in cleanup, umount ROOT/overlay to avoid leaking mounts
- system tests:
- fix a few badly-written tests that assumed/hardcoded overlay
- buildx test: add weird exception to device-number test
- mount tests: add special case code for vfs
- unprivileged test: disable one section that is N/A on vfs
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #20318: now that sqlite is the podman default,
enforce that in CI as well. Test boltdb only in Prior Fedora.
In the process, discovered & cleaned up some duplication
and unused YAML anchors.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The `v4.3.1` version of the library defines a common
`passthrough_envars()` so it doesn't need to be duplicated in podman and
buildah CI. It also includes an update to build-push which should make
debugging easier.
Finally, these images include setting of the en_US.UTF-8 locale to enable
removal of a workaroud in a future commit.
Signed-off-by: Chris Evich <cevich@redhat.com>
When you run e2e tests locally they use CNI unless the NETWORK_BACKEND
env was set to netavark. Because our main focus is on netavark we should
test it by default.
For local tests this should help to prevent CNI/netavark conflicts as I
assume most systems where people run tests on are on netavark by now.
For CI testing we hardcode NETWORK_BACKEND there to test both netavark
(on current fedora) and CNI (prior fedora). MAke sure to switch the
logic in the CI setup to reflect that.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Every few months we get a new CI failure that requires
scanning through logs that give no indication of what
is happening or where. Tracking down the error can
cost many hours.
Solution: pepper cirrus scripts with showrun(), which echoes
the command _and_ displays the source filename + lineno.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Rather than supporting a special-mode *just* for netavark/aardvark
testing [in podman CI], support testing with all the latest
`podman-next` COPR packages.
The idea here is very similar to the netavark/aardvark special mode it
replaces. Most podman-dependencies do not have the level of
comprehensive CI as exist here. This new CI-mode allows testing
upstream updates to podman-dependencies without needing to roll out
a whole new package/release for them.
Also update documentation for this new mode.
Signed-off-by: Chris Evich <cevich@redhat.com>
Porting them over to v2 requires a full rewrite.
IT is not clear who actually uses these benchmarks, Valentin who wrote
them originally is in favor of removing them. He recommends to use
script from hack/perf instead.
This commit also drop the CI integration, it is not clear who actually
uses this data. If it is needed for something please speak up.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Also remove disused `gitlab` test setup. This test was disabled a
while ago and is unlikely to ever be revived.
Signed-off-by: Chris Evich <cevich@redhat.com>
The `benchmarks.env` file is intended for machine consumption.
Including things like a `kB` unit label (like `$MEMTOTAL`) make items
difficult to parse. Additionally, multi-value keys (like `$UNAME_RM`)
make extra/unnecessary work for the interpreter. Simplify these items
and include a data-schema version marker so an interpreter can be made
aware/support future format changes.
Signed-off-by: Chris Evich <cevich@redhat.com>
The `localbenchmarks()` function stores a `.env` file containing current
environment variables for benchmark-classification purposes. However its
naked use of `printenv` means it was logging the contents of library
functions and (worse) trying to stort all the lines. This results in an
unusable mess inside `benchmarks.env`. Fix this by re-using the
purpose-built passthrough_envars() which is designed to only print
useful, safe, env. vars.
Signed-off-by: Chris Evich <cevich@redhat.com>
Output from podman system service, on system tests, is
being saved... it just hasn't been collected as an artifact.
Start collecting it. And, remove obsolete-unused-misleading
code that made me think it _was_ being collected.
Also: log system-service output for bud tests, and set
log-level to info per suggestion from @Luap99
Signed-off-by: Ed Santiago <santiago@redhat.com>
On occasion, developers need to run the latest or bleeding-edge
netavark/aardvark-dns in the podman CI environment. Enable this through
use of magic strings in the PR title, but only if the PR is marked as a
draft. The intent being, when the PR is ready for review, the current
CI VM package versions will be used. Hopefully also reminding the PR
author to remove the magic strings from the title, remove draft status,
and push the final set of changes to be merged.
Also, add documentation about this new magic PR title string and CI's
behaviors.
Signed-off-by: Chris Evich <cevich@redhat.com>
The hardware used for podman-machine testing is fairly
stable/predictable because it's bare-metal. This is a nearly ideal
environment for collection of benchmarking data. Arrange for that to
happen, and the resulting data to be collected.
Also keep track of the benchmark-basis details in a machine-readable
`env` file along side the raw and parsed benchmarks.
Signed-off-by: Chris Evich <cevich@redhat.com>
The passthrough_env function was unnecessarily complicated,
hence fragile. Clean it up, and add regression tests.
For future reference: CI broke horribly because of this.
Rootless tests all failed with missing CI_DESIRED_NETWORK.
Root cause was that CIRRUS_CHANGE_TITLE had a trailing
space which, because of shell indirection, passthrough_env()
wrote as trailing backslash (not backslash-space) in the
/etc/ci_environment file, which then caused the next line
in the file to get glommed onto CIRRUS_CHANGE_TITLE.
Signed-off-by: Ed Santiago <santiago@redhat.com>
We have CI tests running in netavark mode when CNI is desired.
Add a new .cirrus.yml envariable, CI_DESIRED_NETWORK, which
we then force-check in e2e and system tests. Simple copy/paste
of #14912 (the RUNTIME check) with manual s/RUNTIME/NETWORK/
and other minor changes.
Signed-off-by: Ed Santiago <santiago@redhat.com>
add support for podman-remote image scp as well as direct access via the API. This entailed
a full rework of the layering of image scp functions as well as the usual API plugging and type creation
also, implemented podman image scp tagging. which makes the syntax much more readable and allows users t tag the new image
they are loading to the local/remote machine:
allow users to pass a "new name" for the image they are transferring
`podman tag` as implemented creates a new image im `image list` when tagging, so this does the same
meaning that when transferring images with tags, podman on the remote machine/user will load two images
ex: `podman image scp computer1::alpine computer2::foobar` creates alpine:latest and localhost/foobar on the remote host
implementing tags means removal of the flexible syntax. In the currently released podman image scp, the user can either specify
`podman image scp source::img dest::` or `podman image scp dest:: source::img`. However, with tags this task becomes really hard to check
which is the image (src) and which is the new tag (dst). Removal of that streamlines the arg parsing process
Signed-off-by: Charlie Doern <cdoern@redhat.com>
* Replace "setup", "lookup", "cleanup", "backup" with
"set up", "look up", "clean up", "back up"
when used as verbs. Replace also variations of those.
* Improve language in a few places.
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
Most were simply deleted, the main one addressed is in the
"pre-testing" `ext_svc_check.sh` script. It will now verify
accessibility of several key test images we maintain in `quay.io`.
Signed-off-by: Chris Evich <cevich@redhat.com>
Add support for new automation library version w/ `$DEBUG` fix
(ref: containers/automation_images#128) and added definitions
for commonly used Distro/version variables.
Signed-off-by: Chris Evich <cevich@redhat.com>
The sshd service is guaranteed to be running by the VM image build
process - it's required by the packer tool for access. Remove the
startup and check on the sshd service.
For many tests, man ssh connections to/from $ROOTLESS_USER on the
host are needed. To facilitate this, the localhost key is added to
`known_hosts` for root and `$ROOTLESS_USER`. Simplify this setup using
the `ssh-keyscan` tool.
Signed-off-by: Chris Evich <cevich@redhat.com>
There are several runtime contexts (rootless and container) where
`$DEST_BRANCH` is needed but was not supplied. A prior commit
(c486576717) removed the default value, `main` which was being
set incorrectly when CI ran on release branches. Fix this by ensuring
the variable is non-empty upon entry to `setup_environment.sh`, then
ensure it gets passed through to child environments by way of the
`/etc/ci_environment` file. This will maintain compatibility with
both CI and `hack/get_ci_vm.sh` use.
Signed-off-by: Chris Evich <cevich@redhat.com>
The test-skipping optimization is failing as rootless on non-main,
because $DEST_BRANCH is not set. Solution: check for envariable,
skip test if missing. (This was part of my original PR, but was
accidentally removed in #14013)
Also: DEST_BRANCH was silently being defaulted to 'main' in lib.sh.
Remove that: per @cevich, it is no longer necessary.
Fixes: #14131
Signed-off-by: Ed Santiago <santiago@redhat.com>
Newer versions of git are much more pedantic about who owns the
repository files. When setting up to run rootless, prior to this
commit, the repo. ownership was changed from root. This causes
all subsequent git-operations as root to fail:
```
fatal: unsafe repository ('<$GOSRC>' is owned by someone else)
```
Fix this by re-ordering operations, such that the change in ownership is
done immediately before executing as a user. Also disable the
git-ownership check on the source repository assuming the CI environment
is disposable.
Signed-off-by: Chris Evich <cevich@redhat.com>
Normally installing/updating packages at test runtime is highly
discouraged for reliability and efficiency reasons. However, in this
specific case, development work of these packages is still fairly hot.
As a compromise to support podman test development, temporarily update
these two specific packages at runtime. At a future date, when updates
are less frequent, this commit can/should be safely reverted. At that
point, the versions installed at VM image build time will persist.
Signed-off-by: Chris Evich <cevich@redhat.com>
Now that netavark and aardvark are packaged and default in F36, support
CNI-based testing in F35 and Ubuntu.
* Remove the temporary/special `$TEST_ENVIRON=host-netavark` construct.
* Remove dedicated/special integration and system testing tasks.
* Update test-config setup to properly handle CNI vs netavark/aardvark
environments.
* Update package-version logging to operate based on installed packages
(along with some other minor script cleanups).
* Update global environment setup to force `$NETWORK_BACKEND=netavark`
in F36 and later. Except when `upgrade_test` task runs.
* Discontinue installing netavark and aardvark-dns binaries from
upstream build artifacts.
* Drop CGV1-vs-2 policy check. Ubuntu VMs now exclusively test CGv1,
Fedora VMs test CGv2, with F35 testing CNI and F36 testing Netavark.
Signed-off-by: Chris Evich <cevich@redhat.com>
