Issue number: N/A
---------
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
We currently use retries on CI to help catch flaky tests. However,
retries are enabled even when testing locally. This is not ideal because
a failing test will be re-run multiple times before erroring out which
increases the time it takes to run tests locally. I typically write
tests before fixes so I can verify my test is checking the correct
behavior. In this case I don't need it to be re-run twice -- I already
know it's supposed to fail.
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Test retries are only enabled on CI
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
Issue number: N/A
---------
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
In addition to migrating away from `toMatchSnapshot` as found in PRs
such as https://github.com/ionic-team/ionic-framework/pull/28083, I
think it would be valuable to have a lint rule to prevent developers
from accidentally using this API in the future.
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Adds a lint rule which errors if `toMatchSnapshot` is being used in
`*.e2e.ts` files.
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
~Note: This PR's lint step will continue to fail until the remaining
`toHaveScreenshot` migration PRs have been merged. Do note merge this PR
until that has been completed.~ All set!
---------
Co-authored-by: Brandy Carney <brandyscarney@users.noreply.github.com>
Issue number: #
---------
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
Shawn noted that I had an outdated comment in the `playwright.config.ts`
file when working on
https://github.com/ionic-team/ionic-framework/pull/27961:
https://github.com/ionic-team/ionic-framework/pull/27961#pullrequestreview-1572374939
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Remove the old comment
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
Issue number: N/A
---------
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
The team would like to explore solutions for being informed of flaky
tests in a way that is not disruptive to our workflow. Currently, flaky
tests fail immediately which means we have to re-run them every time.
We'd like flaky tests to be automatically retried but also reported to
us so we can address them in a separate PR.
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Enables the Playwright GitHub reporter. This will report about flaky
tests on the PR if applicable as well as in the CI results.
- Enables test retries. Tests will be retried up to 2 times before
failing.
- Disables reporting slow tests in the GitHub reporter. Some of our
tests require gesture interaction which are inherently slow but
otherwise working as intended. We don't necessarily need to know about
these right now.
- Disables "maxFailures". Tests that can fail at most once are never
detected as flaky since they are never retried. As a result, we need to
disable this in order to have flaky tests be reported to us.
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
Issue number: N/A
---------
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
The team has observed that Playwright's default screenshot diff
threshold of 0.2 is not sensitive enough, and screenshots that should be
getting updated are not getting updated.
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Updates the threshold to 0.1
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
---------
Co-authored-by: ionitron <hi@ionicframework.com>
Issue number: N/A
---------
<!-- Please refer to our contributing documentation for any questions on
submitting a pull request, or let us know here if you need any help:
https://ionicframework.com/docs/building/contributing -->
<!-- Some docs updates need to be made in the `ionic-docs` repo, in a
separate PR. See
https://github.com/ionic-team/ionic-framework/blob/main/.github/CONTRIBUTING.md#modifying-documentation
for details. -->
<!-- Please do not submit updates to dependencies unless it fixes an
issue. -->
<!-- Please try to limit your pull request to one type (bugfix, feature,
etc). Submit multiple pull requests if needed. -->
This is the first PR to introduce the infrastructure required to add
test generators to the Ionic Framework project. This PR introduces the
file name changes necessary to support two playwright configs, so I
recommend reviewing the PR by commit:
1e5012cea1
- Created a `playwright.config-legacy.ts` file and updates
`package.json`.
- Running `npm run test.e2e` will run the generator tests, and running
`npm run test.e2e.legacy` will run the legacy tests.
4fe8de7df7
- Updates the GitHub Action scripts to run both the modern and legacy
E2E tests. I added command modifiers to avoid collisions with output
directories.
e8bcfaf926
- Updates `*.e2e.ts` files to have the legacy format name:
`*.e2e-legacy.ts`. This naming scheme is required for the two Playwright
configs to pull in the correct files. When migrating tests to
generators, team members will rename the file to remove the `-legacy`
part.
5bf196c36d
(warning: lots of files!)
- Updates the `*.e2e.ts-snapshots` directories to have the legacy format
name: `*.e2e-legacy.ts-snapshots`. The screenshot directory in
Playwright is generated based on the test file name which is why we are
updating the screenshot directory. When migrating tests to generators,
team members will rename the directory to remove the `-legacy` part.
## Does this introduce a breaking change?
- [ ] Yes
- [x] No
<!-- If this introduces a breaking change, please describe the impact
and migration path for existing applications below. -->
## Other information
<!-- Any other information that is important to this PR such as
screenshots of how the component looks before and after the change. -->
