macOS 14 added new requirements that un-codesigned sandbox apps must be granted access when changed. Waiting for this UI caused macOS tests to fail on macOS 14. Additionally, adding codesigning is not sufficient, since it must still be approved before codesigning is enough to pass the check. As a workaround, this PR disables sandboxing for macOS tests in CI.
https://developer.apple.com/documentation/updates/security#June-2023)
> App Sandbox now associates your macOS app with its sandbox container using its code signature. The operating system asks the person using your app to grant permission if it tries to access a sandbox container associated with a different app. For more information, see [Accessing files from the macOS App Sandbox](https://developer.apple.com/documentation/security/app_sandbox/accessing_files_from_the_macos_app_sandbox).
And that link explains why this is happening on a macOS 14 update:
> In macOS 14 and later, the operating system uses your appâs code signature to associate it with its sandbox container. If your app tries to access the sandbox container owned by another app, the system asks the person using your app whether to grant access. If the person denies access and your app is already running, then it canât read or write the files in the other appâs sandbox container. If the person denies access while your app is launching and trying to enter the other appâs sandbox container, your app fails to launch.
>
> The operating system also tracks the association between an appâs code signing identity and its sandbox container for helper tools, including launch agents. If a person denies permission for a launch agent to enter its sandbox container and the app fails to start, launchd starts the launch agent again and the operating system re-requests access.
Fixes packages part of https://github.com/flutter/flutter/issues/149264.
Verified tests pass:
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20master%20-%20packages/6/overview
The mock process runner used in most of the tests had poor handling of
stdout/stderr:
- By default it would return the `List<int>` output from the mock
process, which was never correct since the parent process runner
interface always sets encodings, thus the `dynamic` should always be
of type `String`
- The process for setting output on a `MockProcess` was awkward, since
it was based on a `Stream<Lint<int>>`, even though in every case what
we actually want to do is just set the full output to a string.
- A hack was at some point added (presumably due to the above issues)
to bypass that flow at the process runner level, and instead return a
`String` set there. That meant there were two ways of setting output
(one of which that only worked for one of the ways of running
processes)
- That hack wasn't updated when the ability to return multiple mock
processes instead of a single global mock process was added, so the
API was even more confusing, and there was no way to set different
output for different processes.
This changes the test APIs so that:
- `MockProcess` takes stdout and stderr as strings, and internally
manages converting them to a `Stream<List<int>>`.
- `RecordingProcessRunner` correctly decodes and returns the output
streams when constructing a process result.
It also removes the resultStdout and resultStderr hacks, as well as the
legacy `processToReturn` API, and converts all uses to the new
structure, which is both simpler to use, and clearly associates output
with specific processes.
To prep for making a combined command to run native tests across different platforms, rework `xctest`:
- Split analyze out into a new `xcode-analyze` command:
- Since the analyze step runs a new build over everything with different flags, this is only a small amount slower than the combined version
- This makes the logic easier to follow
- This allows us to meaningfully report skips, to better notice missing tests.
- Add the ability to target specific test bundles (RunnerTests or RunnerUITests)
To share code between the commands, this extracts a new `Xcode` helper class.
Part of https://github.com/flutter/flutter/issues/84392 and https://github.com/flutter/flutter/issues/86489
