Ref: #16167
According to Cirrus-support, defining and using `$CIRRUS_DEFAULT_WORK`
is responsible for causing a strange race condition on windows. This is
impacting the agent's ability to clone with errors like:
```
Using built-in Git...
Cloning refs/heads/main...
Failed to clone: read ${CIRRUS_DEFAULT_WORK}\.git\HEAD: The process
cannot access the file because another process has locked a portion of
the file.!
```
Fix this by hard-coding a work directory for this windows task.
Co-authored-by: Jason T. Greene <jason@stacksmash.com>
Signed-off-by: Chris Evich <cevich@redhat.com>
As discussed in f2f: this is the cleanest, simplest mechanism
I can think of to auto-test the Big Three dependencies: simply
run go mod edit immediately after git checkout, then run the
entire CI test suite.
This differs significantly from the buildah treadmill, in that
buildah is almost impossible to re-vendor without manual intervention.
(In practice, so are these, but let's dream of a world in which
this will run and pass every night). (I want a pony too).
Signed-off-by: Ed Santiago <santiago@redhat.com>
It's conceivable for CI to spend a lot of time testing code which
otherwise should be rejected due to quality problems. Previously this
was validated in a dedicated task, however a failure would still fail
the CI run. Simplify the number of CI tasks by combining the consistency
check at the tail-end of the build task.
Signed-off-by: Chris Evich <cevich@redhat.com>
Previously, two tasks always ran first, prior to anything else. One to
verify network and external-service connectivity. Another to verify
certain important `.cirrus.yml` standards are met. However, as the
total number of tasks continues to grow, the need to keep these basic
checks as dedicated prerequisites is of decreasing value/importance.
Fold these two checks into a new `pretesting_script` component of the
Fedora `build` task, on both `x86_64` and `aarch64`.
Signed-off-by: Chris Evich <cevich@redhat.com>
There's little practical reason to execute unit-level testing on
multiple platforms, since there's so little platform interaction.
Remove the unit-test runs on Ubuntu, only execute on root-full and
root-less Fedora.
Signed-off-by: Chris Evich <cevich@redhat.com>
There's little need to execute this test for (nearly) every PR.
Further, since it always executes the *latest* upstream tests, there's
no need to run it on any branch other than `main`. Arrange for it to
only execute for the `main` cirrus-cron trigger.
Signed-off-by: Chris Evich <cevich@redhat.com>
The win_installer task fails on the `multiarch` cirrus-cron build.
This is because it depends on the `Windows Cross` (alt_build) task
which is bypassed in this context. This will cause the `repo.tbz`
download to constantly throw 404s. Fix this by skipping the
win_installer task for the `multiarch` (container images) build.
Signed-off-by: Chris Evich <cevich@redhat.com>
We're seeing a huge number of "unknown failure" flakes,
and all seem to be in the 'clone' step, failing with:
echo "$ARTCURL..."
curl ... https://api.cirrus-ci.com/.../repo.tbz
(lots of pretty curl output lines)
curl: (22) The requested URL returned error: 502
Solution: use --retry, which will backoff-retry transient
failures. Start with a backoff of 8s, not the default 1.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Any new files installed by new PRs and those present in unreleased
versions of Podman will need additional manipulation of the
dist-git spec file in the files section to workaround the
`installed but unpackaged files` issue.
The fix-spec-file packit action is useful for this.
The default fix-spec-file action often has trouble guessing the correct
version from upstream code, so it would be beneficial to specify the
correct upstream version as well.
See: https://packit.dev/docs/actions/#fix-spec-file
Rename cirrus task: `Test build RPM` to
`Test build podman-next Copr RPM` for clarity.
[NO NEW TESTS NEEDED]
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
Run machine tests on every PR as label-driven machine test
triggering is currently hard to predict and debug.
Co-authored-by: Ed Santiago <santiago@redhat.com>
Co-authored-by: Miloslav Trmač <mitr@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
AWS EC2 keys VM images by an utterly unreadable, horrible to use,
generated "AMI ID" value. This is very error prone for humans in
practice, since it's impossible to tell one image from the next by
eye. Worse, EC2 permits duplicate name-tag values, complicating
image specification further.
However fortunately, Cirrus-CI recently implemented a feature by
which AMI's may be referenced by a name-tag search - choosing
the most recent AMI found. Since the `containers/automation_images`
build workflow always assigns a unique name + `$IMAGE_SUFFIX` value,
we can simply re-use it for both AWS and GCP image specification.
In other words as of this commit, specifying new CI VM images can
be done by simply updating the `$IMAGE_SUFFIX` value as we've always
done. No need to call out a specific AMI ID just for EC2 tasks.
Signed-off-by: Chris Evich <cevich@redhat.com>
EL8 builds are failing because hack/markdown-preprocess needs python3
which AFAICT isn't included by default in EL8 build environments.
This commit also includes an additional `[CI:COPR]` mode which is
currently runs the same tests as `[CI:DOCS]` but could differ in future.
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
Fixes: #15189
A while ago I updated the only_if/skip options and forgot that several
"cross build" tasks also build documentation in addition to binaries. Re-enable
them to execute all the time, except when Cirrus-cron is making our
multi-arch images.
Signed-off-by: Chris Evich <cevich@redhat.com>
A prior change added extra whitespace when commenting out several
sections to temporarily disable F35 testing. This restores the sections
to proper indentation, so (in the future) only the `#` character needs
to be removed.
Signed-off-by: Chris Evich <cevich@redhat.com>
new file: test/e2e/config_arm64.go
Tests that fail on aarch64 have been skipped with
`skip_if_aarch64`.
Co-authored-by: Chris Evich <cevich@redhat.com>
Co-authored-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
...and enable the at-test-time confirmation, the one that
double-checks that if CI requests runc we actually use runc.
This exposed a nasty surprise in our setup: there are steps to
define $OCI_RUNTIME, but that's actually a total fakeout!
OCI_RUNTIME is used only in e2e tests, it has no effect
whatsoever on actual podman itself as invoked via command
line such as in system tests. Solution: use containers.conf
Given how fragile all this runtime stuff is, I've also added
new tests (e2e and system) that will check $CI_DESIRED_RUNTIME.
Image source: https://github.com/containers/automation_images/pull/146
Since we haven't actually been testing with runc, we need
to fix a few tests:
- handle an error-message change (make it work in both crun and runc)
- skip one system test, "survive service stop", that doesn't
work with runc and I don't think we care.
...and skip a bunch, filing issues for each:
- #15013 pod create --share-parent
- #15014 timeout in dd
- #15015 checkpoint tests time out under $CONTAINER
- #15017 networking timeout with registry
- #15018 restore --pod gripes about missing --pod
- #15025 run --uidmap broken
- #15027 pod inspect cgrouppath broken
- ...and a bunch more ("podman pause") that probably don't
even merit filing an issue.
Also, use /dev/urandom in one test (was: /dev/random) because
the test is timing out and /dev/urandom does not block. (But
the test is still timing out anyway, even with this change)
Also, as part of the VM switch we are now using go 1.18 (up
from 1.17) and this broke the gitlab tests. Thanks to @Luap99
for a quick fix.
Also, slight tweak to #15021: include the timeout value, and
reword message so command string is at end.
Also, fixed a misspelling in a test name.
Fixes: #14833
Signed-off-by: Ed Santiago <santiago@redhat.com>
Instead of requiring developers to search for a magic button, make the
task trigger at the time a special PR label is added. Update comments
accordingly.
Signed-off-by: Chris Evich <cevich@redhat.com>
We're still not testing runc in CI (#14833), and it may be weeks
or months before we can, due to criu/glibc nightmare, but one day
we'll be back on track, then later on we'll update VMs again,
and screw it up, and lose runc, and not notice, and RHEL will
break, and oh noes headless chicken again, repeat repeat.
We can do better. Use .cirrus.yml to explicitly define which
VMs should use which runtimes, and enforce it early in the
CI build step. This should never fail (uh huh) in a PR,
only in one of the update-VM PRs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The podman-machine integration tests are designed to execute on
bare-metal, since they perform significant work with virtual-machines.
This test is costly to run at scale, so it is limited to being manually
triggered by developers (for now). A 'trigger' button will appear in the
task status page of the Github WebUI once all test dependencies are met.
In the Cirrus-CI WebUI, there is also a 'pre-trigger' button that may be
pressed if a developer doesn't wish to wait. Also:
* Add a `localmachine` target in the `Makefile` on the off-chance
developers wish to execute locally. Update the `ginkgo-run` target
to accommodate re-use by the new `localmachine` target.
* Exclude `podman_machine` task from `success` dependency verification.
This also involves adding an exception to `cirrus_yaml_test.py`
otherwise it will complain loudly.
* ***NOTE*** Inclusion of `ec2_instance` in *any* task will cause
`hack/get_ci_vm.sh` to barf and be non-functional. Future updates will
be made to restore functionality. Before then, simply comment out
the `ec2_instance` section as a temporarily workaround.
Signed-off-by: Chris Evich <cevich@redhat.com>
This would've caught a regression that #14549 had to fix.
Let's try to prevent the next regression.
This requires some hackery to get namespaces initialized
before the service is started; otherwise the service itself
initializes namespaces, which basically ends up with a
server process that runs forever.
Also: in stop_service(), reset service_pid, because that's
the correct thing to do.
Also: add some debug statements to try to figure out a
CI failure. (And leave them in place, because they might
be useful for future problems).
Signed-off-by: Ed Santiago <santiago@redhat.com>
When passing artifacts from one task to another, if a direct dependency
was not created due to `only_if`, a race condition can occur: The
descendent task may start running prior to the indirect (but required)
dependency (`build` in this case) completing and uploading artifacts.
Fix this by spamming a `build` dependency into all tasks which require
it (basically everything).
Signed-off-by: Chris Evich <cevich@redhat.com>
Using both the 'skip' and 'only_if' features at the same time may be
hard for maintainers to decipher. Consolidate them into `only_if` since
that bypasses creation of the task all together - meaning there are
potentially fewer tasks for a developer to scroll through.
Since the `multiarch` Cirrus-Cron build no-longer depends on the direct
"build-ability" from the current repo. state, it can be further
optimized. When operating in this context, avoid running many/most
other tasks, depending instead only on `ext_svc_check`.
Finally, add a simple document describing the various runtime contexts
along with the list of expected tasks. Reference this prominently right
in front of every `only_if` so it's impossible for a maintainer to miss.
Signed-off-by: Chris Evich <cevich@redhat.com>
The caching update made in
https://github.com/containers/podman/pull/14016 neglected to attend to
the "git repo." needs of this task. Fix this so images can build.
Also, make this optional task only appear when the `[CI:BUILD]` magic
string is used. No reason to tempt every PR author to press a
mysterious button labeled 'trigger'.
Signed-off-by: Chris Evich <cevich@redhat.com>
GOPROXY's default value is "https://proxy.golang.org,direct"
since go 1.13, so it is redundant to set it explicitly.
For some reason though, GOPROXY in Cirrus CI is set to direct,
which makes things such as go mod tidy very slow. So, set the
proper (default) value for in in .cirrus.yml. Do the same for GOSUMDB.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Remove GOPATH setting as since Go 1.9 it defaults to $HOME/go (for
earlier versions it had to be specified explicitly).
Remove GOPATH-related code from the spec, using relative paths when
compiling packages, and enable Go modules, simplifying the spec.
Remove support for multiple paths in GOPATH (which is rarely used and
doesn't really work with modules).
Remove setting GOBIN, rely on $GOPATH/bin instead. In case GOBIN is
explicitly set (which is highly unlikely), forcefully ignore by
unsetting it.
Remove GOBIN from tools invocation since we added GOPATH/bin to PATH.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Add support for new automation library version w/ `$DEBUG` fix
(ref: containers/automation_images#128) and added definitions
for commonly used Distro/version variables.
Signed-off-by: Chris Evich <cevich@redhat.com>
With the increasingly complex `.cirrus.yml` task relationships, build
cache wasn't always working as intended. Recently, non-build tasks were
observed assuming authority over `gopath_cache`. Ref.:
https://github.com/containers/podman/pull/13998#issuecomment-1108834538
Address this by an overall simplification using artifacts instead of
cache. Using artifacts allows establishing concrete
authorship/authority over cached repo. content. In this way, dependent
tasks may simply consume the artifact with `curl` instead of relying on
complex caching algorithms.
Also/Minor: Add YAML checking to the pre-commit configuration.
Signed-off-by: Chris Evich <cevich@redhat.com>
Now that netavark and aardvark are packaged and default in F36, support
CNI-based testing in F35 and Ubuntu.
* Remove the temporary/special `$TEST_ENVIRON=host-netavark` construct.
* Remove dedicated/special integration and system testing tasks.
* Update test-config setup to properly handle CNI vs netavark/aardvark
environments.
* Update package-version logging to operate based on installed packages
(along with some other minor script cleanups).
* Update global environment setup to force `$NETWORK_BACKEND=netavark`
in F36 and later. Except when `upgrade_test` task runs.
* Discontinue installing netavark and aardvark-dns binaries from
upstream build artifacts.
* Drop CGV1-vs-2 policy check. Ubuntu VMs now exclusively test CGv1,
Fedora VMs test CGv2, with F35 testing CNI and F36 testing Netavark.
Signed-off-by: Chris Evich <cevich@redhat.com>
Newer versions of git (like `2.35`) fail on certain operations (like
`rebase` and `am`) without a local identity. Add a fake one from the
start, with a clearly identifiable test-value to avoid problems at
runtime.
Signed-off-by: Chris Evich <cevich@redhat.com>
