745 Commits

Author SHA1 Message Date
2b86ab5a1e Change image-path to image for tests
As we deprecate image-path for image.

Signed-off-by: Ashley Cui <acui@redhat.com>
2024-02-29 13:38:53 -05:00
527b3793b8 Use machine image as specified in containers.conf
For podman machine init, deprecate the --image-path option for --image.
--image now accepts the correct image from containers.conf

Also, add the ability to specify an OCI image from the --image flag using the docker:// transport.

Signed-off-by: Ashley Cui <acui@redhat.com>
2024-02-29 13:38:43 -05:00
96f9d0867a Merge pull request #21854 from ashley-cui/locks
Move locks to shim layer
2024-02-29 14:34:03 +00:00
8377483145 Merge pull request #21866 from cgwalters/prefix-errpull
machine/ociartifact: Include image name in error
2024-02-29 11:32:05 +00:00
12694d5424 Merge pull request #21592 from l0rd/compress-refactoring-v5
Refactor machine decompress.go
2024-02-29 10:09:36 +00:00
35133c8424 Move locks to shim layer
Previously, the locks were on the provider layer, which doesn't make a vm operation with a config file update atomic. Move them up a layer, so the entire function locks while doing provider and config operations.

This adds a Remove and a Set function to the shim layer.

[NO NEW TESTS NEEDED] Unsure how to test this

Signed-off-by: Ashley Cui <acui@redhat.com>
2024-02-29 00:12:04 -05:00
fd1d951262 Move ignition functions into Containerfiles
We used to use ignition to perform any customization required for podman
machine because our input was a generic FCOS image.  Now that we are
building our own images, some of this customization can be migrated to
the Containerfile itself and be less of a burden in our code at boot up.

At the time of this PR, the Containerfile can be found at
https://github.com/baude/podman-machine-images/tree/main.  It is only
present for a so-called daily image.  There is little liklihood that
this would the final location for the Containerfile so consider it a
working version only.

Split WSL and rest apart in the e2e tests so we no longer ppull the
generic FCOS image for testing.

Note: the change to the pull image name is so PRs are not immediately
broken that are already in the queue.

[NO NEW TESTS REQUIRED]

Signed-off-by: Brent Baude <bbaude@redhat.com>
2024-02-28 16:05:50 -06:00
b58f20bdb6 machine/ociartifact: Include image name in error
If we hit an error here, it will be really useful to know

- That we're trying to fetch a container image
- Which image we're trying to fetch

[NO NEW TESTS NEEDED]

Signed-off-by: Colin Walters <walters@verbum.org>
2024-02-28 16:23:34 -05:00
8f02822c12 Remove xz unit tests as they are flaky
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-28 15:37:15 +01:00
c93ed35b09 Merge pull request #21848 from baude/machinestartcleanup
Clean up gvproxy if machine start fails
2024-02-28 12:08:21 +00:00
f8abd7ff75 machine config: make write atomic
As indicated in #21849, loading the machine config can flake/fail with
an EOF JSON error indicating an incomplete file.  Address the issue by
atomically writing the config.  This way, it is not possible to load an
incomplete or partially written file.  The lock can be acquired later on
to sync state.

[NO NEW TESTS NEEDED] as it's a hard-to-hit race.

Fixes: #21849
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
2024-02-28 09:44:41 +01:00
9d1704396a Merge pull request #21572 from mheon/hyperv_9p_mkdir
Ensure HyperV 9p mounts work when a dir doesn't exist
2024-02-27 22:34:07 +00:00
26ec570c65 Ensure HyperV 9p mounts work when a dir doesn't exist
Before, we required that the mount target exist and be a
directory for the 9p mount to successfully complete, which is not
how things are supposed to work - the user should be able to
mount anywhere. This should just be a simple mkdir, but with FCOS
the root directory is immutable so we need to undo that before we
can mkdir, and unfortunately we don't have a library that can do
chattr (and I didn't want to drag in a new dependency just for
that), so let's be gross and add it to the SSH command. I
aggressively dislike this but it does work.

[NO NEW TESTS NEEDED] Can worry about getting a more generic
mount test together for Machine later.

Signed-off-by: Matt Heon <mheon@redhat.com>
2024-02-27 14:33:53 -05:00
dfae51a4b8 Clean up gvproxy if machine start fails
Matt found a bug where if a machine start did not run to completion, a
gvproxy was left around running.  This gvproxy then subsequently stopped
the next attempt to start.

Signed-off-by: Brent Baude <bbaude@redhat.com>

[NO NEW TESTS NEEDED]
2024-02-27 13:33:43 -06:00
7b6d9a586e Fixup: avoid overriding io.Copy error
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-27 16:48:47 +01:00
59704665ae Avoid overriding io.Copy error
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-27 16:16:44 +01:00
24a33a538c Move code for sparse optimized copy to a dedicated method
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-27 15:59:56 +01:00
bed61806c8 pkg/machine/ocipull: add default policy.json file
So that this file can be inculded in our windows/macos packages and also
by other packagers.
Right now the default policy is allow everything but we plan to add
signing in the future.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-02-27 15:29:59 +01:00
a02aa8f6a2 pkg/machine/ocipull: add custom policy.json location
The default policy file /etc/containers/policy.json location does not
work on windows and for packages that ship a default.

Now we search for the policy.json in the following overwrite locations:
macos and linux:
 - ~/.config/containers/policy.json
 - /etc/containers/policy.json
windows:
 - %APPDATA%\containers\policy.json

Also it offers an additional DefaultPolicyJSONPath var that should be
overwritten at built time with the path of the file that is shipped by
packagers. Thile file is used when none of the overwrite paths exist.

[NO NEW TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-02-27 15:29:58 +01:00
04f70320c1 Merge pull request #21594 from arixmkii/qemu-win-machine-5
Change QEMU netdev to Unix domain socket
2024-02-26 20:37:06 +00:00
656bf7b764 Change QEMU netdev to Unix domain socket
This change migrates to new QEMU stream netdev added in 7.2.0.
It also unifies how gvproxy is used in QEMU and AppleHV.

Signed-off-by: Arthur Sengileyev <arthur.sengileyev@gmail.com>
2024-02-26 20:02:46 +02:00
19b676f855 Enable lint for Darwin and fix identified issues
[NO NEW TESTS NEEDED] Just fixing lint issues

Signed-off-by: Matt Heon <mheon@redhat.com>
2024-02-26 11:01:54 -05:00
f099250beb Better file close and err handling
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:03 +01:00
ea4553d590 integrating changes from #21768
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
fa99b9be93 renamed testfiles as testdata
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
0b3f3f0ef1 Use faster gzip reader
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
2245cf8dc4 Remove duplication and make consistent usage of the progress bar
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
c42d3a74ed Add a comment to explain why we look at file name for zip files
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
95aa2c4e68 Use github.com/stretchr/testify assert in compression_test.go
Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
88af8852db Refactor machine decompress.go
Added some tests to verify that files extractions works
with different compression format.

Created a decompressor interface with 2 main methods:
  reader(): returns an io.Reader for the specific compression algorithm
  copy(): extracts the compressed file into the file provided as param

Created 5 decompressor types:
- gzip: extract gzip files
- xz: extract xz files
- zip: extract zip files
- generic: extract any other file using github.com/containers/image/v5/pkg/compression
- uncompressed: only do a copy of the file

Minor fix to the progress bar instances: added a call to bar.Abort(false)
that happens before Progress.Wait() to avoid that it hangs when a bar is
not set as completed although extraction is done.

Signed-off-by: Mario Loriedo <mario.loriedo@gmail.com>
2024-02-25 22:29:02 +01:00
1fc9d5ae6e Merge pull request #21797 from mtrmac/sparse0
Reformulate sparseWriter
2024-02-25 14:36:08 +00:00
eabce71cd2 Merge pull request #21804 from n1hility/ready-race
Fix race conditions in hyperv backend
2024-02-24 14:07:38 +00:00
b68d3c7a0e Fix race conditions in hyperv readiness checking
- Listen before starting the vm
- Fix a device race caused by lazy hv_vsock init by waiting on network manager

[NO NEW TESTS NEEDED]

Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
2024-02-24 01:43:28 -06:00
5d303ca267 Reformulate sparseWriter to deal with starting/ending zeroes explicitly
... instead of using a multi-variable state machine.

The net effect of this code is exactly the same as the previous implementation,
except:
- the operation after Write() returns an error might differ
- If the file ends with zeroes, we don't Seek(-1), and
  we don't create a hole at all if it is too small, preferring
  to save a syscall.

But this formulation is hopefully easier to prove correct.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-02-23 02:09:39 +01:00
c5434bf711 Expand sparseWriter tests
- Add more test cases
- Test that we create the expected (large) holes;
  don't enforce anything for the <zerosThresholt ones.

O)nly changes test code, should not change behavior.

Signed-off-by: Miloslav Trmač <mitr@redhat.com>
2024-02-23 02:09:39 +01:00
36d8e27601 Merge pull request #21517 from jakecorrenti/fix-qemu-todos
machine: Address some QEMU TODOs
2024-02-22 21:21:50 +00:00
b338364fbf Merge pull request #21786 from Luap99/machine-gvproxy-cleanup
pkg/machine: ignore gvproxy pidfile not exists error
2024-02-22 16:13:41 +00:00
5a844511c8 Merge pull request #21652 from Luap99/machine-http-proxy
machine: implement http proxy logic for all providers
2024-02-22 15:40:39 +00:00
d7f7f07c9b Stop using rm'd inspect format templates in tests
Fixes the "machine rm --save-ignition --save-image" test so that it no longer
uses the `{{ .Image }}` format string.

Fixes the "init should cleanup on failure" test so that it no longer
uses the `{{ .Image }}` and `{{ .ConfigPath }}` format strings.

Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
2024-02-22 08:34:41 -05:00
be9aba7ccd Remove image provenance from machine inspect
We don't care about the provenance of the machine image, so this is no
longer applicable to have when displaying info.

Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
2024-02-22 08:34:27 -05:00
d68240b150 Update machine files rm order and add QEMU rm
Changes the order in which the machine-specific files are removed in
`Remove()`. Removes the system connections first, then removes the
`configPath` last. `configPath` is removed last, because in the case of
an error with any of the previous files, the removal can be attempted
again since the machine still "exists".

Made the errors in `Remove` hard errors instead of soft errors.

Added the implementation for the QEMU-specific file removal.

[NO NEW TESTS NEEDED]

Signed-off-by: Jake Correnti <jakecorrenti+github@proton.me>
2024-02-22 08:31:55 -05:00
6f6925cca4 pkg/machine: ignore gvproxy pidfile not exists error
When gvproxy exits it will delete the pidfile itself so we need to
account for that and juts ignore the case, it just means gvproxy was
able to exit successfully on its own.

Also remove the useless defer and return the error so we can get an
error exit code not just a print on stderr.

Currently it shows this error which is not helpful to any user:
unable to clean up gvproxy: "unable to read gvproxy pid file /run/user/1000/podman/gvproxy.pid: open /run/user/1000/podman/gvproxy.pid: no such file or directory"

[NO NEW TESTS NEEDED] TODO: make machine tests check stderr for such
things.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-02-22 12:39:20 +01:00
669e718561 Merge pull request #21636 from jakecorrenti/inspectinfo-hostuser
machine: Remove unnecessary TODOs
2024-02-22 10:29:51 +00:00
f218f8430a machine: implement http proxy logic for all providers
Copy all proxy envs into the VM on each start, this allows for updates
without having to recrate the VM. This is implemented via shell script
that is passed via ssh to the VM after it is started.

With that we now use the same logic for all providers the old fw_cfg
logic for qemu has been removed and the WSL code as well which keeps the
behavior the same.

There is a small risk now because we only update the env via ssh that
processes started before will have the old incorrect env but it should
really only effect core system processes which likely do not need them
anyway. The podman system service should not be started at this point
so it should be good enough.

It also fixes the broken behavior with SSL_CERT_FILE/SSL_CERT_DIR which
were updated on each start which is not correct as the files are only
copied with ignition so these should not be updated and just set
statically when the VM was created.

e2e test has been added to ensure the behavior works as expected.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-02-22 11:23:45 +01:00
59b6f48d90 Merge pull request #21735 from jakecorrenti/inspect-conn-vals
machine: Add `ConnectionInfo` to inspect
2024-02-22 02:36:50 +00:00
0d1a8c0f2b Merge pull request #21768 from baude/zstd
zstd now default compression for podman machine
2024-02-21 14:46:38 +00:00
f756e5db68 Merge pull request #21747 from mheon/windows_lint
Fix Lint on Windows and enable the job
2024-02-21 00:47:04 +00:00
a31e8d2a23 zstd now default compression for podman machine
given that we are moving to building our own machine images, we have
decided to use zstd compression as it is superior in speed to the
alternatives.  as such, this pr adds zstd to our machine code; and also
has to account for dealing with sparseness on darwin; which the default
zstd golang library does not.

[NO NEW TESTS NEEDED]

Signed-off-by: Brent Baude <bbaude@redhat.com>
2024-02-20 14:26:41 -06:00
1d4651bccc Merge pull request #21763 from giuseppe/sparse-file
machine: add sparse file writer
2024-02-20 19:35:33 +00:00
d9c706ea95 Merge pull request #21694 from arixmkii/wait-for-gvproxy
Extract waitForGvProxy into shared utility function
2024-02-20 18:59:46 +00:00