Simply because it's been a while since the last testimage
build, and I want to confirm that our image build process
still works.
Added /home/podman/healthcheck. This saves us having to
podman-build on each healthcheck test. Removed now-
unneeded _build_health_check_image helper.
testimage: bump alpine 3.16.2 to 3.19.0
systemd-image: f38 to f39
- tzdata now requires dnf **install**, not reinstall
(this is exactly the sort of thing I was looking for)
PROBLEMS DISCOVERED:
- in e2e, fedoraMinimal is now == SYSTEMD_IMAGE. This
screws up some of the image-count tests (CACHE_IMAGES).
- "alter tarball" system test now barfs with tar < 1.35.
TODO: completely replace fedoraMinimal with SYSTEMD_IMAGE
in all tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
- tmpfs + noswap test: requires noswap feature in kernel.
Check for it, and skip if unimplemented. (Root only.
Rootless test works regardless of kernel).
- podman generate systemd tests: always use --files option,
because otherwise the "DEPRECATED" warning gets written
to the systemd unit file.
- kube play tests: yikes. Fix longstanding bugs when checking
for containers running. This revealed a longstanding bug
in one test: multi-pod YAML never actually worked. Fixed now.
- run_podman(): that new check-for-warnings code we added
in #19878, duh, I skipped it on Debian but should've skipped
when *runc*. Do so now and update the comment. Requires
minor surgery to podman_runtime() helper to avoid
infinite recursion.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Our test registry (used for login & local registry tests)
was being run using the standard podman tmpdir, hence the
standard podman database, This was then getting clobbered
in the 330-corrupt-images test, which runs "system reset".
We just didn't know this was happening. Until we added
a registry test after the system reset. Oops.
Solution: new helper function podman_isolation_opts()
sets --root, --runroot, *and --tmpdir*. Refactor all
existing --root/--runroot usages. Document.
Next problem: the "network reload" test in 500-networking.bats
did not (could not) know about our registry port, so the
"iptables -F" command reverted that to DROP, so the subsequent
podman-auth in 700-play timed out.
Solution: add a podman-isolated "network reload" to start_registry().
Final problem, because, really, those weren't enough: a BATS
bug where running with --filter-tags would set IFS=',' in setup_suite
which in turn has catastrophic consequences:
https://github.com/bats-core/bats-core/issues/812
See #20966 for details of the failure and further conversation.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Very rare flake, probably caused by my nemesis, podman run -d
Solution: keep the sleep-1 (vs using nanosecond resolution),
but make sure we first wait for the output from the container.
Also, bump down the iteration delay in wait_for_output, from 5s to 1.
Thanks to Paul for noticing that.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #20797 (defer assertion failures). The bail-now()
helper was being defined only in setup() ... and some tests,
particularly 001-basic.bats, define their own minimalist setup().
Symptom was "bail-now: command not found", which still caused
test to fail (so no failures were hidden) but led to concern
and wasted time when analyzing failures.
Solution: add one more definition of bail-now(), in outer scope.
There is still one pathological case I'm not addressing: a
bats file that defines its own teardown() which does not invoke
basic_teardown(), then has a test that runs defer-assertion-failures
without a followup immediate-assertion-failures. This would lead
to failures that are never seen. Since teardown() without basic_teardown()
is invalid, I choose not to worry about this case.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add support for .pod unit files with only PodmanArgs, GlobalArgs, ContainersConfModule and PodName
Add support for linking .container units with .pod ones
Add e2e and system tests
Add to man page
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
Some system tests run deep loops:
for x in a b c; do
for y in d e f; do
.... check condition $x + $y
Normally, if one of these fails, game over. This can be frustrating
to a developer looking for failure patterns.
Here we introduce a new defer-assertion-failure function, meant
to be called before loops like these. Everything is the same,
except that tests will continue running even after failure.
When test finishes, or if test runs immediate-assertion-failure,
a new message indicates that multiple tests failed:
FAIL: X test assertions failed. Search for 'FAIL': above this line.
Signed-off-by: Ed Santiago <santiago@redhat.com>
We're only testing vfs in CI. That's bad. #18822 tried to
remedy that but that only worked on system tests, not e2e.
Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml
in the same vein as all the other CI_DESIRED_X. Since it's 2023
we default to overlay, testing vfs only in priorfedora.
Fixes required:
- e2e tests:
- in cleanup, umount ROOT/overlay to avoid leaking mounts
- system tests:
- fix a few badly-written tests that assumed/hardcoded overlay
- buildx test: add weird exception to device-number test
- mount tests: add special case code for vfs
- unprivileged test: disable one section that is N/A on vfs
Signed-off-by: Ed Santiago <santiago@redhat.com>
This is something I've long wanted in logs: an indicator of
which bats file the test lives in. As of v1.7.0 there is
now a way to do that, BATS_TEST_NAME_PREFIX. Use it. Logs
now look like:
ok 14 [001] podman - shutdown engines
ok 15 [005] podman info - basic test
...
not ok 195 [065] podman cp - dot notation ....
(As a bonus, we can remove the super-long "test blah blah pasta"
duplication from 505.bats).
Also, removed no-longer-necessary (fingers crossed) debug code
for the recently fixed containers-storage umount/EINVAL flake.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #20394. For years (since BATS 1.5) we've been
seeing and ignoring nasty red warnings at the end of every
system test run. Thanks for fixing it, @giuseppe! But it
broke down in the '?' case when $expected_rc is empty:
test/system/helpers.bash: line 345: [: -eq: unary operator expected
Simple fix.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Part of RUN-1906.
Followup to #19878 (check stderr in system tests): allow_warnings()
and require_warning() functions to make sure no unexpected messages
fall through the cracks.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Followup to #20016:
- remove obsolete (misleading) comment
- prune dangling <none>:<none> image
Also, in kube test, rmi pause_image to avoid nasty red warnings
Also, ouch, fix a stupid that I introduced in #19878: the PODMAN
command path got dropped from log messages.
Signed-off-by: Ed Santiago <santiago@redhat.com>
With few exceptions, commands that exit 0 should not emit any
messages with level=warning or =error. Let's start enforcing
that in run_podman.
Allow one-off exceptions, typically when we're testing an
actual warning condition (usual case: "podman stop" where it
times out to SIGKILL). Exceptions are specified via:
run_podman 0+w subcommand...
^^^---- or, rarely, 0+e
"0" stands for "expect exit status 0", which is the default
so it's implicit anyway. The +w / +e (or even +we) is the
new part. I have added it to tests where necessary.
And, because life is what it is, add two global exceptions:
- Debian. Because runc has too many flakes.
- kube. Ditto. Kube commands emit lots of nasty error
messages (yes, level=error) that don't seem to affect
results.
Similar to #18442
Signed-off-by: Ed Santiago <santiago@redhat.com>
Unexplained infrequent flakes in sdnotify system tests,
waiting for READY=1.
Hypothesis: race condition between the container sending
the READY string and that string making it through conmon
and socat into the log file.
Solution: don't just check once; keep trying in a loop.
Write a reusable wait_for_file_content() helper function,
and clean up a bunch more tests as long as we're at it.
Fixes: #19724
Signed-off-by: Ed Santiago <santiago@redhat.com>
- the "podman {run,exec} /etc" test: runc now spits out
"is a directory" instead of "permission denied". And,
on exec, exits 255 instead of 126. Deal with it.
- workaround for https://github.com/containers/skopeo/issues/823
(skopeo XDG bug): always make sure XDG is defined for skopeo
Signed-off-by: Ed Santiago <santiago@redhat.com>
To silence my find-obsolete-skips script, remove the '#'
from the following issues in skip messages:
#11784#15013#15025#17433#17436#17456
Also update the messages to reflect the fact that the issues
will never be fixed.
Also remove ubuntu skips: we no longer test ubuntu.
Also remove one buildah skip that is no longer applicable:
Fixes: #17520
Signed-off-by: Ed Santiago <santiago@redhat.com>
The podman-login tests have accumulated much cruft over the
years, because that's the only place where we run a local
registry, and the process was crufty: we actually start/stopped
the registry as the first & last tests of the file. Meaning,
you couldn't do 'hack/bats 150:just-one-test' because that
would skip the registry start. And just now, a completely
unrelated test has had to be shoved into the login file.
This PR revamps the whole thing, by adding a new registry helper
module that can be used anywhere. And, once the registry is
started, it just stays running until the end of tests. (This
requires BATS 1.7 or greater).
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add new _prefetch helper for fetching and caching images.
Use it in a few places, most importantly 120-load.bats
where our teardown() now runs 'rmi -af'.
Reason: in #17911 we discovered that podman save + load do
not actually preserve the image: annotations and other metadata
are lost. This means that a test which runs after 120-load.bats
is operating on a different $IMAGE than a test which runs before.
This is not a problem except in very obscure corner cases, like
one fixed in #18542, but it seems irresponsible to just handwave
that issue away
The _prefetch function uses skopeo for fetching and saving
images, because skopeo preserves digests and metadata.
[Side note for posterity: I tried amending basic_setup() to
always rmi -a + prefetch, instead of the current images -a +
rmi unwanted ones. That slowed down system tests by 10 minutes,
presumably because loads are much slower than queries. I reverted
that change and am documenting it as a reminder of why we do things
the way we do.]
Signed-off-by: Ed Santiago <santiago@redhat.com>
for #18514: if we get a timeout in teardown(), run and show
the output of podman system locks
for #18831: if we hit unmount/EINVAL, nothing will ever work
again, so signal all future tests to skip.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The current way of bind mounting the host timezone file has problems.
Because /etc/localtime in the image may exist and is a symlink under
/usr/share/zoneinfo it will overwrite the targetfile. That confuses
timezone parses especially java where this approach does not work at
all. So we end up with an link which does not reflect the actual truth.
The better way is to just change the symlink in the image like it is
done on the host. However because not all images ship tzdata we cannot
rely on that either. So now we do both, when tzdata is installed then
use the symlink and if not we keep the current way of copying the host
timezone file in the container to /etc/localtime.
Also note that we need to rebuild the systemd image to include tzdata in
order to test this as our images do not contain the tzdata by default.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2149876
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Instrument system tests in hopes of tracking down #17216,
the unlinkat-ebusy-hosed flake.
Oh, also, timestamp.awk: timestamps have always been UTC, but
add a 'Z' to make it unambiguous.
Signed-off-by: Ed Santiago <santiago@redhat.com>
In run_podman(), display a nanosecond-level timestamp next to
each command and its output.
Because this clutters the results, teach logformatter to grok
these new timestamps, strip them, and display a more human-readable
time delta in the left-hand timestamp column. logformatter started off
as a mess and is now, well, 🤮. I'm sorry. I just hope its results
make it worthwhile.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Several tweaks to see if we can track down #17216, the unlinkat-ebusy
flake:
- teardown(): if a cleanup command fails, display it and its
output to the debug channel. This should never happen, but
it can and does (see #18180, dependent containers). We
need to know about it.
- selinux tests: use unique pod names. This should help when
scanning journal logs.
- many tests: add "-f -t0" to "pod rm"
And, several unrelated changes caught by accident:
- images-commit-with-comment test: was leaving a stray image
behind. Clean it up, and make a few more readability tweaks
- podman-remote-group-add test: add an explicit skip()
when not remote. (Otherwise, test passes cleanly on
podman local, which is misleading)
- lots of container cleanup and/or adding "--rm" to run commands,
to avoid leaving stray containers
Signed-off-by: Ed Santiago <santiago@redhat.com>
- fix a typo that was resulting in a test being a NOP, and
add actual testing to it.
- fix two Expects() with incorrectly-ordered actual/expects
- remove leading whitespace from an It() test name
- To(BeTrue()) is evil. Wherever possible, replace it with
useful string or field checks. When not possible, use
the annotation field to indicate what failed. I got
carried away here, #sorrynotsorry
- remove unused system-test code
Signed-off-by: Ed Santiago <santiago@redhat.com>
On cgroup v1 we need to mount only the systemd named hierarchy as
writeable, so we configure the OCI runtime to mount /sys/fs/cgroup as
read-only and on top of that bind mount /sys/fs/cgroup/systemd.
But when we use a private cgroupns, we cannot do that since we don't
know the final cgroup path.
Also, do not override the mount if there is already one for
/sys/fs/cgroup/systemd.
Closes: https://github.com/containers/podman/issues/17727
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
...safer, too: the big change is using 'mapfile' to split
multiline strings; this preserves empty lines, making it
easy to see spurious (or missing) blank lines in output.
Another change is to indent the expected-output string
consistently, for readability.
Then, to handle \r (CR) and other control characters, use
bash %q to format special chars. But %q makes\ it\ hard\ to
read\ lines\ with\ spaces, so strip off those backslashes.
This makes assert() much larger and uglier, but this is
code that shouldn't be touched often.
Finally, because these are big changes to critical code,
write a complicated regression test suite for assert().
Signed-off-by: Ed Santiago <santiago@redhat.com>
Tests constantly fail with zero indication of why. Fix that.
- add correct default for $QUADLET path
- add check to make sure it exists
- log quadlet commands and their output
Signed-off-by: Ed Santiago <santiago@redhat.com>
Red Hat registry is too unreliable. (As of this writing
in January 2023, quay.io is not much better, but this is
a new flake. Ubi has been flaking for a year or more).
Instead of UBI, use the new systemd-image added to system tests
in #16814. Since this reduces the number of cached images,
a few unrelated tests (image count) need to be tweaked.
And, sigh, Fedora systemd colorizes boot messages by default,
causing a failure where we don't see an expected Reached Target
message. I don't want to rely on ASCII formatting codes, so
I've updated the build-systemd-image script so it disables
systemd colors, and have built a new systemd-image:20230106.
Made a few small usability improvements to the script as well.
Closes: #16695
Signed-off-by: Ed Santiago <santiago@redhat.com>
...based on f37, not f31. And make it fedora-minimal so it's
smaller. And clean up dnf so it's even smaller. And tag it
with our proper YMD tag, and commit the script that builds it.
This broke the system-df tests. In the process of resolving
that, I found those tests a little lacking. So, improve their
coverage a little bit.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This adds basic container and volume system tests for quadlet. These
install and run actual systemd units and ensure they work.
Signed-off-by: Alexander Larsson <alexl@redhat.com>
The 900-ssh test is not an actual test, and I'm unable to
figure out how to make it one. Skip it for now, but add a
bunch of FIXMEs some someone can come in later and actually
implement it.
Also removed lots of dead code and misleading comments.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The main helpers.bash file is rather bloated and it's difficult to
find stuff there. Move networking functions to their own helper
file.
While at it, apply a consistent style, and rearrange logically
related functions into sections.
Suggested-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Using bash /dev/tcp/ pseudo-device files to probe for bound ports has
indeed the advantage of simplicity, but comes with a few drawbacks:
- it will actually send data to unsuspecting services that might be
running in the same network namespace as the tests, possibly
causing unwanted interactions
- it doesn't allow for UDP probing
- it makes it impossible to clearly distinguish between different
address bindings
Replace that approach with a new helper, port_is_bound(), that uses
procfs entries at /proc/net to detect bound ports, without the need
for active probing.
We can now implement optional parameters in callers, to check if a
port if free for binding to a given address, including any IPv4
(0.0.0.0) or any IPv6 (::0) address, and for a given protocol, TCP
or UDP.
Extend random_free_port() and random_free_port_range() to support
that.
The implementation of one function in the file
test/system/helpers.bash, namely ipv6_to_procfs(), and the
implementation of the corresponding own test, delimited by the
markers "# BEGIN ipv6_to_procfs" and "# END ipv6_to_procfs" in the
file test/system/helpers.c was provided, on the public forum at:
https://github.com/containers/podman/pull/16141
by Ed Santiago <santiago@redhat.com>, who expressly invited me to
include them in this code submission.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Currently, wait_for_port() duplicates the check logic implemented by
port_is_free().
Add an optional argument to port_is_free(), representing the bound
address to check, and call it, dropping the direct check in
wait_for_port().
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
It looks like #16132 was my fault: a missing 'wait' for a container
to exit. Let's see if this fixes the flake.
And, while poking through flake logs, I found another missing wait.
And... in wait_for_output(), address a potential race.
Signed-off-by: Ed Santiago <santiago@redhat.com>
One of the system tests was creating a volume and not cleaning up
after itself. Fix that: do cleanup in the test itself. And, add
a 'volume rm -af' to global teardown() to leave things clean for
the next tests.
Also, OOPS! Correct some instances of 'podman' in two system
tests to 'run_podman'. And remove an unused (misleading) variable.
And, one more: in auto-update test, unit file, use $PODMAN,
not /usr/bin/podman
UGH! Yet one more: found/fixed a 'run<space>podman'
Signed-off-by: Ed Santiago <santiago@redhat.com>
PR #16141 introduces a new network type, "pasta". Its tests
rely on running 'ip -j' and socat in the container. Add them.
Also: bump to alpine 3.16.2 (from 3.16.0)
Also: clean up apk cache, this saves us 2MB+ in the image
Also (unrelated): clean up two broken uses of '$(< ...)' that
are causing tests to blow up under bats 1.8 on my laptop
New testimage is 20221018 and, sigh, is 12.7MB (up 4MB).
Signed-off-by: Ed Santiago <santiago@redhat.com>
Trying to catch the wiley metacopy flake: add a debug
condition to run_podman, in system tests, to log all
instances in which output includes the metacopy warning.
The idea is to detect the very first time it happens,
and see what is triggering it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
For systems that have extreme robustness requirements (edge devices,
particularly those in difficult to access environments), it is important
that applications continue running in all circumstances. When the
application fails, Podman must restart it automatically to provide this
robustness. Otherwise, these devices may require customer IT to
physically gain access to restart, which can be prohibitively difficult.
Add a new `--on-failure` flag that supports four actions:
- **none**: Take no action.
- **kill**: Kill the container.
- **restart**: Restart the container. Do not combine the `restart`
action with the `--restart` flag. When running inside of
a systemd unit, consider using the `kill` or `stop`
action instead to make use of systemd's restart policy.
- **stop**: Stop the container.
To remain backwards compatible, **none** is the default action.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
This exposed a nasty bug in our system-test setup: Ubuntu (runc)
was writing a scratch containers.conf file, and setting CONTAINERS_CONF
to point to it. This was well-intentionedly introduced in #10199 as
part of our long sad history of not testing runc. What I did not
understand at that time is that CONTAINERS_CONF is **dangerous**:
it does not mean "I will read standard containers.conf and then
override", it means "I will **IGNORE** standard containers.conf
and use only the settings in this file"! So on Ubuntu we were
losing all the default settings: capabilities, sysctls, all.
Yes, this is documented in containers.conf(5) but it is such
a huge violation of POLA that I need to repeat it.
In #14972, as yet another attempt to fix our runc crisis, I
introduced a new runc-override mechanism: create a custom
/etc/containers/containers.conf when OCI_RUNTIME=runc.
Unlike the CONTAINERS_CONF envariable, the /etc file
actually means what you think it means: "read the default
file first, then override with the /etc file contents".
I.e., we get the desired defaults. But I didn't remember
this helpers.bash workaround, so our runc testing has
actually been flawed: we have not been testing with
the system containers.conf. This commit removes the
no-longer-needed and never-actually-wanted workaround,
and by virtue of testing the cap-drops in kube generate,
we add a regression test to make sure this never happens
again.
It's a little scary that we haven't been testing capabilities.
Also scary: this PR requires python, for converting yaml to json.
I think that should be safe: python3 'import yaml' and 'json'
works fine on a RHEL8.7 VM from 1minutetip.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Integrate sd-notify policies into `kube play`. The policies can be
configured for all contianers via the `io.containers.sdnotify`
annotation or for indidivual containers via the
`io.containers.sdnotify/$name` annotation.
The `kube play` process will wait for all containers to be ready by
waiting for the individual `READY=1` messages which are received via
the `pkg/systemd/notifyproxy` proxy mechanism.
Also update the simple "container" sd-notify test as it did not fully
test the expected behavior which became obvious when adding the new
tests.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
implement new ssh interface into podman
this completely redesigns the entire functionality of podman image scp,
podman system connection add, and podman --remote. All references to golang.org/x/crypto/ssh
have been moved to common as have native ssh/scp execs and the new usage of the sftp package.
this PR adds a global flag, --ssh to podman which has two valid inputs `golang` and `native` where golang is the default.
Users should not notice any difference in their everyday workflows if they continue using the golang option. UNLESS they have been using an improperly verified ssh key, this will now fail. This is because podman was incorrectly using the
ssh callback method to IGNORE the ssh known hosts file which is very insecure and golang tells you not yo use this in production.
The native paths allows for immense flexibility, with a new containers.conf field `SSH_CONFIG` that specifies a specific ssh config file to be used in all operations. Else the users ~/.ssh/config file will be used.
podman --remote currently only uses the golang path, given its deep interconnection with dialing multiple clients and urls.
My goal after this PR is to go back and abstract the idea of podman --remote from golang's dialed clients, as it should not be so intrinsically connected. Overall, this is a v1 of a long process of offering native ssh, and one that covers some good ground with podman system connection add and podman image scp.
Signed-off-by: Charlie Doern <cdoern@redhat.com>
new file: test/e2e/config_arm64.go
Tests that fail on aarch64 have been skipped with
`skip_if_aarch64`.
Co-authored-by: Chris Evich <cevich@redhat.com>
Co-authored-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
* Correct spelling and typos.
* Improve language.
Co-authored-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
Wrong variable. And, wrong index range. And, wrong bash
syntax for extracting end_port. And, add explicit check
for valid range, because die() inside 'foo=$(...)' will not
actually die. And, refactor some confusing code. And,
reformat/clean up a confusing and too-wide comment.
Fixes: #14854
Signed-off-by: Ed Santiago <santiago@redhat.com>
The test must ensure that all ports in the range are free not just
the first. This flakes often because port 5355 is always in use by
systemd-resolved on fedora.
Fixes#14716
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
