opensource/grafana
1
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2026-03-13 15:29:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

CI: Speed up Playwright e2e tests workflow (#119277)

Browse Source 
* CI: Replace Dagger builds with native make for Playwright e2e tests

Switch from Dagger-based builds to native Go/JS builds for the
Playwright e2e test pipeline. Grafana now runs as a native binary
on the CI runner instead of in a Docker container.

Key changes:
- build-backend: actions/setup-go + make build-go (instead of Dagger)
- build-frontend: actions/setup-node + make build-js + yarn e2e:plugin:build
- run-playwright-tests: downloads artifacts, uses start-server script
  to run Grafana natively (instead of Docker container from GHCR)
- build-grafana: standalone full Dagger build, off the Playwright
  critical path (still produces Docker/tarball for push-docker-image
  and run-a11y-test)
- required-playwright-tests: no longer depends on build-grafana
- Remove debug env vars (ACTIONS_STEP_DEBUG, RUNNER_DEBUG)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix zizmor template-injection: use docker/login-action for GHCR login in Playwright job

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace GHA service container with docker run for Grafana

GHA service containers start before any step (including checkout), so
volume-mounted config files don't exist yet and Grafana crashes. The
health check never passes, blocking all steps from running.

Switch to docker run -d in a step after checkout, so all files are
available when the container starts. This eliminates the need for the
docker restart workaround and the zizmor unpinned-images suppression.

Verified locally: built all three Dagger steps (backend, frontend,
assembly with --import-dir + chmod +x), loaded the Docker image, and
confirmed Grafana starts successfully with volume-mounted config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix Docker image binary permissions lost by actions/upload-artifact

actions/upload-artifact strips execute permissions (all files become 644).
The backend binaries need +x restored before Dagger packages them into the
Docker image, otherwise the container fails with "Permission denied" when
trying to exec the grafana binary.

Verified locally: pulled the CI-built image from GHCR, confirmed binaries
had 664 permissions, added chmod +x, and tested the full service container
restart flow successfully.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix CI failures: pin docker/login-action, fix docker tarball glob, suppress zizmor unpinned-images

- Pin docker/login-action@v3 to hash @5e57cd118135c172c3672efd75eb46360885c0ef
- Use glob *.docker.tar.gz in push-docker-image (Dagger produces versioned filenames)
- Add unpinned-images ignore for pr-e2e-tests.yml (dynamic build output image)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CI: Fix missing bundled-plugins directory in build-grafana

actions/upload-artifact skips empty directories, so the bundled-plugins
dir (empty in OSS builds) doesn't exist after download. Create it before
running Dagger to prevent the --import-dir from failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CI: Speed up Playwright e2e tests workflow

Split the monolithic Grafana build into three Dagger jobs (backend, frontend, assembly) with granular caching. Use the --import-dir flag to pre-populate the artifact store, skipping compilation in the assembly step. Run Playwright shards in parallel with 4 workers instead of 1, reduced from 8 to 6 shards, and use GHA service containers with bind-mounted config instead of building custom e2e Docker images. Add workflow concurrency, job timeouts, and dependency caching. This reduces critical path from ~32 minutes to ~17 minutes on cold builds and ~9 minutes with warm caches.

Expected impact:
- Parallel backend/frontend builds save 6-8 minutes (vs sequential)
- GHA output cache hits reduce builds to 0 seconds on cache hit
- Docker service container approach eliminates per-shard overhead (5-7 min saved)
- 4 workers per shard and reduced retry count improve test throughput
- Workflow concurrency prevents wasted runs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

try merging frontend artifact

run the tests

shard tree artifact, delete artifacts, fix ts files being excluded

copy bin

fix path

fix path

fix script

another try

fix incorrect permissions

* try stitching together standalone grafana build

* include more dirs in frontend build

fix paths

* try caching node-modules better

try caching node-modules

disable YARN_ENABLE_HARDENED_MODE

temporarily stop caching node_modules to test performance

temp don't cache node_modules to measure perf

fix frontend cache

* add script for downloading the report and viewing it locally

* Update codeowners

* Add workflow to build grafana docker image

* add placeholder check

* Use hosted runners for everything

* Bump actions versions

* Don't cache playwright browser installs

* build e2e test plugins in each shard

* Split bench report into seperate step

and update bench to v1

* try packaging less of the public dir

* Package up whole public directory

its needed for some reason

* Run the grafana server migrations in the background while playwright installs

* Fix flaky time picker preferences tests

* Fix detect-changes always running e2e tests

* Skip building frontend source maps

* Don't check out repo in report steps

* Add per-shard failure instructions
This commit is contained in:
Josh Hunt
2026-03-05 12:54:49 +00:00
committed by GitHub
parent 91b6ee64c8
commit 23c6c1e50b
8 changed files with 387 additions and 377 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/CODEOWNERS vendored
View File

@@ -1096,6 +1096,7 @@ playwright.storybook.config.ts @grafana/grafana-frontend-platform
/scripts/levitate-parse-json-report.js @grafana/grafana-frontend-platform
/scripts/levitate-show-affected-plugins.js @grafana/grafana-frontend-platform
/scripts/codemods/explicit-barrel-imports.cjs @grafana/frontend-ops
/scripts/view-playwright-ci.sh @grafana/grafana-frontend-platform
/scripts/codeowners-manifest/ @grafana/dataviz-squad
/scripts/test-coverage-by-codeowner.js @grafana/dataviz-squad
@@ -1332,6 +1333,7 @@ embed.go @grafana/grafana-as-code
/.github/workflows/pr-test-docker.yml @grafana/grafana-developer-enablement-squad
/.github/workflows/update-schema-types.yml @grafana/grafana-frontend-platform
/.github/workflows/defaults-ini-docs-reminder.yml @grafana/docs-tooling @jtvdez
/.github/workflows/pr-build-grafana.yml @grafana/grafana-developer-enablement-squad
# Generated files not requiring owner approval
/packages/grafana-data/src/types/featureToggles.gen.ts @grafanabot

2
.github/actions/change-detection/action.yml vendored
View File

@@ -24,7 +24,7 @@ outputs:
description: Whether the e2e tests or self have changed in any way
value: ${{ steps.changed-files.outputs.e2e_any_changed == 'true' ||
steps.changed-files.outputs.backend_any_changed == 'true' ||
steps.changed-files.outputs.frontend_any_changed == 'true' || 'true' }}
steps.changed-files.outputs.frontend_any_changed == 'true' }}
e2e-cloud-plugins:
description: Whether the cloud plugins code or tests have changed in any way
value: ${{ steps.changed-files.outputs.e2e_cloud_plugins_any_changed || 'true' }}

129
.github/workflows/pr-build-grafana.yml vendored Normal file
View File

@@ -0,0 +1,129 @@
name: PR build Docker image
on:
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
permissions: {}
jobs:
detect-changes:
# Gate the whole workflow behind this check
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
name: Detect whether code changed
runs-on: ubuntu-x64-small
permissions:
contents: read
outputs:
changed: ${{ steps.detect-changes.outputs.frontend == 'true' || steps.detect-changes.outputs.backend == 'true' }}
steps:
- uses: actions/checkout@v5
with:
persist-credentials: true # required to get more history in the changed-files action
fetch-depth: 2
- name: Detect changes
id: detect-changes
uses: ./.github/actions/change-detection
with:
self: .github/workflows/pr-build-grafana.yml
build-grafana:
needs: detect-changes
if: needs.detect-changes.outputs.changed == 'true'
name: Build & Package Grafana
runs-on: ubuntu-x64-xlarge
permissions:
contents: read
id-token: write
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Build Grafana
uses: kminehart/dagger-for-github@590bdefcb7f0c1030fce068a2c041c36582262d6
env:
CACHE_BUILDERS: ${{ github.event.pull_request.head.repo.fork == false }}
with:
cache-binary: "true"
version: 0.18.8
verb: run
args: go run ./pkg/build/cmd artifacts -a docker:grafana:linux/amd64 --grafana-dir="${PWD}" --cache-builders=${CACHE_BUILDERS} > out.txt
- name: Cat built artifact
run: cat out.txt
- name: Move built artifacts
run: |
mkdir -p build-dir
mv "$(grep 'grafana_.*docker.tar.gz$' out.txt)" build-dir/grafana.docker.tar.gz
- name: Set artifact name
run: echo "artifact=grafana-server-${{github.run_number}}" >> "$GITHUB_OUTPUT"
id: artifact
- name: Upload grafana docker tarball
uses: actions/upload-artifact@v7
with:
retention-days: 1
name: grafana-docker-tar-gz
path: build-dir/grafana.docker.tar.gz
push-docker-image:
needs:
- detect-changes
- build-grafana
if: needs.detect-changes.outputs.changed == 'true'
name: Push PR Docker image
permissions:
contents: read
id-token: write
runs-on: ubuntu-x64
steps:
- id: get-github-token
name: "create github app token"
uses: grafana/shared-workflows/actions/create-github-app-token@eb02241ed0a92aff205feab8ac3afcdf51c757c8 # create-github-app-token-v0.2.0
with:
github_app: "delivery-bot-app"
- uses: grafana/shared-workflows/actions/login-to-gar@main
id: login-to-gar
with:
registry: 'us-docker.pkg.dev'
environment: 'dev'
- uses: actions/download-artifact@v8
with:
name: grafana-docker-tar-gz
path: .
- name: Load & Push Docker image
env:
BUILD_ID: ${{ github.run_id }}
run: |
set -euo pipefail
LOADED_IMAGE_NAME=$(docker load -i grafana.docker.tar.gz | sed 's/Loaded image: //g')
VERSION=$(echo "${LOADED_IMAGE_NAME}" | cut -d ':' -f 2 | cut -d '-' -f 1)
DOCKER_IMAGE="us-docker.pkg.dev/grafanalabs-dev/docker-grafana-dev/grafana:${VERSION}-${BUILD_ID}"
docker tag "${LOADED_IMAGE_NAME}" "${DOCKER_IMAGE}"
docker push "${DOCKER_IMAGE}"
echo "IMAGE=${DOCKER_IMAGE}" >> "$GITHUB_ENV"
- name: Add PR status check
env:
GH_TOKEN: ${{ steps.get-github-token.outputs.token }}
SHA: ${{ github.event.pull_request.head.sha }}
run: |
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
/repos/grafana/grafana/check-runs \
-f "name=${IMAGE}" \
-f "head_sha=${SHA}" \
-f 'status=completed' \
-f 'conclusion=neutral' \
-f 'output[title]=Docker image' \
-f "output[summary]=${IMAGE}" \
-f "output[text]=${IMAGE}"

599
.github/workflows/pr-e2e-tests.yml vendored
View File

@@ -7,30 +7,25 @@ on:
- main
- release-*.*.*
# TODO: re-enable this before merging
# concurrency:
# group: ${{ github.workflow }}-${{ github.ref }}
# cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
permissions: {}
env:
ACTIONS_STEP_DEBUG: true
RUNNER_DEBUG: 1
jobs:
detect-changes:
# Run on `grafana/grafana` main branch, or on pull requests to prevent double-running on mirrors
if: (github.event_name == 'pull_request' || (github.event_name == 'push' && github.repository == 'grafana/grafana'))
name: Detect whether code changed
runs-on: ubuntu-latest
runs-on: ubuntu-x64-small
permissions:
contents: read
outputs:
changed: ${{ steps.detect-changes.outputs.e2e }}
cloud_plugins_changed: ${{ steps.detect-changes.outputs.e2e-cloud-plugins }}
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v6
with:
persist-credentials: true # required to get more history in the changed-files action
fetch-depth: 2
@@ -40,240 +35,98 @@ jobs:
with:
self: .github/workflows/pr-e2e-tests.yml
build-grafana:
build-backend:
needs: detect-changes
if: needs.detect-changes.outputs.changed == 'true'
name: Build & Package Grafana
name: Build backend
runs-on: ubuntu-x64-xlarge
timeout-minutes: 15
permissions:
contents: read
id-token: write
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
- name: Checkout repo
uses: actions/checkout@v6
with:
persist-credentials: false
# TODO: add a cleanup workflow to remove the cache when the PR is closed
# https://github.com/actions/cache/blob/main/tips-and-workarounds.md#force-deletion-of-caches-overriding-default-cache-eviction-policy
# TODO: maybe we could just use the cache to store the build, instead of uploading as an artifact?
- uses: actions/cache@v4
- name: Cache backend build
uses: actions/cache@v5
id: cache
with:
key: "build-grafana-${{ runner.os }}-${{ hashFiles('yarn.lock', 'public/*', 'packages/*', 'conf/*', 'pkg/**/*.go', '**/go.mod', '**/go.sum', '!**_test.go', '!**.test.ts', '!**.test.tsx', 'Dockerfile') }}"
path: |
build-dir
key: "e2e-build-backend-${{ runner.os }}-${{ hashFiles('**/go.mod', '**/go.sum', 'go.work', 'go.work.sum', 'pkg/**/*.go', 'apps/**/*.go', '!**_test.go', 'Makefile', 'embed.go') }}"
path: bin
# If no cache hit, build Grafana
- name: Build Grafana
- name: Setup Go
uses: actions/setup-go@v6.3.0
if: steps.cache.outputs.cache-hit != 'true'
uses: kminehart/dagger-for-github@590bdefcb7f0c1030fce068a2c041c36582262d6
env:
CACHE_BUILDERS: ${{ github.event.pull_request.head.repo.fork == false }}
with:
cache-binary: "true"
version: 0.18.8
verb: run
args: go run ./pkg/build/cmd artifacts -a targz:grafana:linux/amd64 -a docker:grafana:linux/amd64 --grafana-dir="${PWD}" --cache-builders=${CACHE_BUILDERS} > out.txt
- name: Cat built artifact
go-version-file: 'go.mod'
cache: true
- name: Build backend
if: steps.cache.outputs.cache-hit != 'true'
run: cat out.txt
- name: Move built artifacts
if: steps.cache.outputs.cache-hit != 'true'
run: |
mkdir -p build-dir
mv "$(grep 'grafana_.*tar.gz$' out.txt | grep -Fv -m1 'docker')" build-dir/grafana.tar.gz
mv "$(grep 'grafana_.*docker.tar.gz$' out.txt)" build-dir/grafana.docker.tar.gz
run: make build-go
# If cache hit, validate the artifact is present
- name: Validate artifact
if: steps.cache.outputs.cache-hit == 'true'
run: |
if [ ! -f build-dir/grafana.tar.gz ]; then
echo "Error: build-dir/grafana.tar.gz not found in cache"
exit 1
fi
- name: Set artifact name
run: echo "artifact=grafana-server-${{github.run_number}}" >> "$GITHUB_OUTPUT"
id: artifact
- name: Upload grafana.tar.gz
uses: actions/upload-artifact@v5
- name: Upload backend build artifacts
uses: actions/upload-artifact@v7
with:
name: grafana-backend
path: bin/
retention-days: 1
name: grafana-tar-gz
path: build-dir/grafana.tar.gz
- name: Upload grafana docker tarball
uses: actions/upload-artifact@v5
with:
retention-days: 1
name: grafana-docker-tar-gz
path: build-dir/grafana.docker.tar.gz
# TODO: we won't need this when we only have playwright
build-e2e-runner:
build-frontend:
needs: detect-changes
if: needs.detect-changes.outputs.changed == 'true'
name: Build E2E test runner
runs-on: ubuntu-latest
name: Build frontend
runs-on: ubuntu-x64-xlarge
timeout-minutes: 15
permissions:
contents: read
id-token: write
outputs:
artifact: ${{ steps.artifact.outputs.artifact }}
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
- name: Checkout repo
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Setup Go
uses: actions/setup-go@v6.0.0
with:
go-version-file: go.mod
cache: ${{ !github.event.pull_request.head.repo.fork }}
- name: Build E2E test runner
id: artifact
run: |
set -euo pipefail
# We want a static binary, so we need to set CGO_ENABLED=0
CGO_ENABLED=0 go build -o ./e2e-runner ./e2e/
echo "artifact=e2e-runner-${{github.run_number}}" >> "$GITHUB_OUTPUT"
- uses: actions/upload-artifact@v5
id: upload
with:
retention-days: 1
name: ${{ steps.artifact.outputs.artifact }}
path: e2e-runner
push-docker-image:
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
needs:
- build-grafana
steps:
- id: get-github-token
name: "create github app token"
uses: grafana/shared-workflows/actions/create-github-app-token@eb02241ed0a92aff205feab8ac3afcdf51c757c8 # create-github-app-token-v0.2.0
- name: Cache frontend build
uses: actions/cache@v5
id: cache
with:
github_app: "delivery-bot-app"
- uses: grafana/shared-workflows/actions/login-to-gar@main
id: login-to-gar
with:
registry: 'us-docker.pkg.dev'
environment: 'dev'
- uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53
with:
name: grafana-docker-tar-gz
path: .
- name: Load & Push Docker image
env:
BUILD_ID: ${{ github.run_id }}
run: |
set -euo pipefail
LOADED_IMAGE_NAME=$(docker load -i grafana.docker.tar.gz | sed 's/Loaded image: //g')
VERSION=$(echo "${LOADED_IMAGE_NAME}" | cut -d ':' -f 2 | cut -d '-' -f 1)
DOCKER_IMAGE="us-docker.pkg.dev/grafanalabs-dev/docker-grafana-dev/grafana:${VERSION}-${BUILD_ID}"
docker tag "${LOADED_IMAGE_NAME}" "${DOCKER_IMAGE}"
docker push "${DOCKER_IMAGE}"
echo "IMAGE=${DOCKER_IMAGE}" >> "$GITHUB_ENV"
- name: Add PR status check
env:
GH_TOKEN: ${{ steps.get-github-token.outputs.token }}
SHA: ${{ github.event.pull_request.head.sha }}
run: |
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
/repos/grafana/grafana/check-runs \
-f "name=${IMAGE}" \
-f "head_sha=${SHA}" \
-f 'status=completed' \
-f 'conclusion=neutral' \
-f 'output[title]=Docker image' \
-f "output[summary]=${IMAGE}" \
-f "output[text]=${IMAGE}"
key: "e2e-build-frontend-${{ runner.os }}-${{ hashFiles('yarn.lock', 'package.json', 'packages/**', 'public/**', 'scripts/webpack/**', '!**/*.test.ts', '!**/*.test.tsx', 'tsconfig.json', '.browserslistrc', 'nx.json', 'project.json', '.nvmrc') }}"
path: |
public/build
public/app/plugins/*/*/dist
run-e2e-tests:
needs:
- build-grafana
- build-e2e-runner
strategy:
fail-fast: false
matrix:
include:
- suite: various-suite (old arch)
path: e2e/old-arch/various-suite
flags: --flags="--env dashboardScene=false"
- suite: dashboards-suite (old arch)
path: e2e/old-arch/dashboards-suite
flags: --flags="--env dashboardScene=false"
- suite: smoke-tests-suite (old arch)
path: e2e/old-arch/smoke-tests-suite
flags: --flags="--env dashboardScene=false"
- suite: panels-suite (old arch)
path: e2e/old-arch/panels-suite
flags: --flags="--env dashboardScene=false"
name: ${{ matrix.suite }}
runs-on: ubuntu-x64-large
permissions:
contents: read
id-token: write
- name: Setup Node.js
if: steps.cache.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-node
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
with:
persist-credentials: false
- uses: actions/download-artifact@v6
with:
name: grafana-tar-gz
- uses: actions/download-artifact@v6
with:
name: ${{ needs.build-e2e-runner.outputs.artifact }}
- name: chmod +x
run: chmod +x ./e2e-runner
- name: Run E2E tests
uses: kminehart/dagger-for-github@590bdefcb7f0c1030fce068a2c041c36582262d6
with:
cache-binary: "true"
version: 0.18.8
verb: run
args: go run ./pkg/build/e2e --package=grafana.tar.gz
--suite=${{ matrix.path }}
${{ matrix.flags }}
- name: Set suite name
id: set-suite-name
if: success() || failure()
- name: Install yarn dependencies
if: steps.cache.outputs.cache-hit != 'true'
uses: ./.github/actions/yarn-install
- name: Build frontend
if: steps.cache.outputs.cache-hit != 'true'
env:
SUITE: ${{ matrix.path }}
run: |
set -euo pipefail
echo "suite=$(echo "$SUITE" | sed 's/\//-/g')" >> "$GITHUB_OUTPUT"
- uses: actions/upload-artifact@v5
if: success() || failure()
NO_SOURCEMAP: '1' # Don't generate sourcemaps for CI builds to speed up webpack + packaging
run: make build-js
- name: Upload frontend build artifacts
uses: actions/upload-artifact@v7
with:
name: ${{ steps.set-suite-name.outputs.suite }}-${{ github.run_number }}
path: videos
name: grafana-frontend
path: |
public/
!**/*.ts
!**/*.tsx
!public/**/node_modules/
!public/locales/
retention-days: 1
run-storybook-test:
name: Verify Storybook (Playwright)
runs-on: ubuntu-latest
runs-on: ubuntu-x64
timeout-minutes: 15
needs: detect-changes
if: needs.detect-changes.outputs.changed == 'true'
permissions:
@@ -287,32 +140,31 @@ jobs:
if: github.event.pull_request.head.repo.fork == false
- name: Checkout code
uses: actions/checkout@v5
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
uses: ./.github/actions/setup-node
- name: Install dependencies
run: yarn install --immutable
- name: Install yarn dependencies
uses: ./.github/actions/yarn-install
- name: Install Playwright browsers
run: npx playwright install --with-deps
run: yarn playwright install chromium --with-deps
- name: Run Storybook and E2E tests
run: yarn e2e:playwright:storybook
run-playwright-tests:
needs:
- build-grafana
- build-backend
- build-frontend
name: Playwright E2E tests (${{ matrix.shard }}/${{ matrix.shardTotal }})
runs-on: ubuntu-x64-large
timeout-minutes: 20
permissions:
contents: read
id-token: write
strategy:
fail-fast: false
@@ -321,129 +173,122 @@ jobs:
shardTotal: [8]
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/download-artifact@v6
- name: Setup Node.js
uses: ./.github/actions/setup-node
- name: Install yarn dependencies
uses: ./.github/actions/yarn-install
# It's quicker to build them in each test shard then prebuild and upload/download them as artifacts
- name: Build test plugins
run: yarn e2e:plugin:build
# Download the grafana artifacts and stitch them together into a standalone prod-ish Grafana distribution
- name: Download backend artifacts
uses: actions/download-artifact@v8
with:
name: grafana-tar-gz
- name: Run E2E tests
uses: kminehart/dagger-for-github@590bdefcb7f0c1030fce068a2c041c36582262d6
name: grafana-backend
path: grafana-server/bin
- name: Fix binary permissions
run: chmod +x grafana-server/bin/grafana*
- name: Download frontend artifacts
uses: actions/download-artifact@v8
with:
cache-binary: "true"
version: 0.18.8
verb: run
args: go run ./pkg/build/e2e-playwright --package=grafana.tar.gz --shard=${{ matrix.shard }}/${{ matrix.shardTotal }} --blob-dir=./blob-report
- uses: actions/upload-artifact@v5
name: grafana-frontend
path: grafana-server/public
- name: Copy misc server files
run: |
cp -r conf/ grafana-server
cp -r tools/ grafana-server
# Apply E2E test configuration on top of the production-mirrored distribution
- name: Configure for E2E testing
run: |
mkdir -p grafana-server/conf/provisioning/{datasources,dashboards,alerting,plugins}
cp scripts/grafana-server/custom.ini grafana-server/conf/custom.ini
cp devenv/datasources.yaml grafana-server/conf/provisioning/datasources/
cp devenv/dashboards.yaml grafana-server/conf/provisioning/dashboards/
cp devenv/alert_rules.yaml grafana-server/conf/provisioning/alerting/
cp devenv/plugins.yaml grafana-server/conf/provisioning/plugins/
mkdir -p grafana-server/data/plugins
cp -r e2e-playwright/test-plugins/. grafana-server/data/plugins/
# Start Grafana from the standalone directory
# The server is immediately backgrounded so it can initialise in parallel while the Playwright browsers are installing 🤓
- name: Start Grafana server
run: |
grafana-server/bin/grafana server \
--homepath="$PWD/grafana-server" \
cfg:server.http_port=3001 \
> grafana-server/grafana.log 2>&1 &
printf "\nThe Grafana server is starting in the background.\n"
echo "To view the server logs, wait for this job to complete and downloaded the 'grafana-server-log-${{ matrix.shard }}' artifact."
- name: Install Playwright browsers
run: yarn playwright install chromium --with-deps
- name: Wait for Grafana server to start
run: |
timeout=180
elapsed=0
printf "Waiting for Grafana to start"
while ! curl -s -f http://localhost:3001/health > /dev/null 2>&1; do
if [ $elapsed -ge $timeout ]; then
printf "\nTimeout after %d seconds waiting for Grafana\n" "$timeout"
exit 1
fi
printf "."
sleep 1
elapsed=$((elapsed + 1))
done
printf "\nThe Grafana server is ready\n"
echo "To view the server logs, wait for this job to complete and downloaded the 'grafana-server-log-${{ matrix.shard }}' artifact."
- name: Run Playwright tests
run: yarn e2e:playwright --shard=${{ matrix.shard }}/${{ matrix.shardTotal }} --reporter=dot,blob --output=playwright-test-results
env:
GRAFANA_URL: http://localhost:3001
CI: true
- name: Show failure instructions
if: failure()
env:
RUN_ID: ${{ github.run_id }}
run: |
echo "Tests in this shard failed."
echo ""
echo "View combined logs and HTML report in the 'All Playwright tests complete' job once all tests finish."
exit 1
- uses: actions/upload-artifact@v7
if: always()
with:
name: grafana-server-log-${{ matrix.shard }}
path: grafana-server/grafana.log
retention-days: 7
- uses: actions/upload-artifact@v7
if: success() || failure()
with:
name: playwright-blob-${{ github.run_number }}-${{ matrix.shard }}
path: ./blob-report
name: playwright-blob-${{ matrix.shard }}
path: blob-report
retention-days: 1
run-azure-monitor-e2e:
if: needs.detect-changes.outputs.cloud_plugins_changed == 'true' && github.event.pull_request.head.repo.fork == false && github.event_name == 'pull_request'
runs-on: ubuntu-x64-large
needs:
- build-grafana
- detect-changes
permissions:
contents: read
id-token: write
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
with:
persist-credentials: false
- uses: grafana/shared-workflows/actions/login-to-gar@login-to-gar-v0.4.0
id: login-to-gar
with:
registry: "us-docker.pkg.dev"
environment: "dev"
- id: pull-docker-image
run: |
docker pull us-docker.pkg.dev/grafanalabs-dev/docker-oss-plugin-partnerships-dev/e2e-playwright:latest
- id: vault-secrets
uses: grafana/shared-workflows/actions/get-vault-secrets@main
with:
repo_secrets: |
AZURE_SP_APP_ID=cpp-azure-resourcemanager-credentials:application_id
AZURE_SP_PASSWORD=cpp-azure-resourcemanager-credentials:application_secret
AZURE_TENANT=cpp-azure-resourcemanager-credentials:tenant_id
- id: deploy-resources
env:
AZURE_SP_APP_ID: ${{ env.AZURE_SP_APP_ID}}
AZURE_SP_PASSWORD: ${{ env.AZURE_SP_PASSWORD}}
AZURE_TENANT: ${{ env.AZURE_TENANT }}
NAME: ${{ github.ref_name }}
run: |
docker container run --name cpp-e2e-deploy -e AZURE_SP_APP_ID -e AZURE_SP_PASSWORD -e AZURE_TENANT -e PLAYWRIGHT_CI=true us-docker.pkg.dev/grafanalabs-dev/docker-oss-plugin-partnerships-dev/e2e-playwright:latest ./cpp-e2e/scripts/ci-run-playwright.sh azure "${NAME}" deploy
- id: extract-creds
# see https://github.com/grafana/oss-plugin-partnerships/blob/a77040d0456003cd258668b61d542dc7c75db5b5/e2e/scripts/deploy.sh#L25 for path
run: |
docker cp cpp-e2e-deploy:/outputs.json /tmp/outputs.json
- uses: actions/download-artifact@v6
with:
name: grafana-tar-gz
- name: Run E2E tests
uses: kminehart/dagger-for-github@590bdefcb7f0c1030fce068a2c041c36582262d6
with:
cache-binary: "true"
version: 0.18.8
verb: run
args: go run ./pkg/build/e2e-playwright --package=grafana.tar.gz --playwright-command="yarn e2e:playwright:cloud-plugins" --cloud-plugin-creds=/tmp/outputs.json
- name: Destroy resources
if: always() && steps.deploy-resources.outcome == 'success'
env:
AZURE_SP_APP_ID: ${{ env.AZURE_SP_APP_ID }}
AZURE_SP_PASSWORD: ${{ env.AZURE_SP_PASSWORD }}
AZURE_TENANT: ${{ env.AZURE_TENANT }}
NAME: ${{ github.ref_name }}
run: |
docker container run --name cpp-e2e-destroy -e AZURE_SP_APP_ID -e AZURE_SP_PASSWORD -e AZURE_TENANT us-docker.pkg.dev/grafanalabs-dev/docker-oss-plugin-partnerships-dev/e2e-playwright:latest ./cpp-e2e/scripts/ci-run-playwright.sh azure "${NAME}" destroy
required-playwright-tests:
needs:
- run-playwright-tests
- run-azure-monitor-e2e
- run-storybook-test
- build-grafana
if: ${{ !cancelled() }}
name: All Playwright tests complete
runs-on: ubuntu-latest
runs-on: ubuntu-x64-small
permissions:
contents: read
id-token: write
steps:
- uses: grafana/shared-workflows/actions/dockerhub-login@main
if: github.event.pull_request.head.repo.fork == false
- uses: grafana/shared-workflows/actions/login-to-gar@main
if: github.event.pull_request.head.repo.fork == false
- uses: actions/checkout@v5
with:
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
- name: Download blob reports from GitHub Actions Artifacts
uses: actions/download-artifact@v6
with:
@@ -464,35 +309,17 @@ jobs:
- name: Merge into HTML Report
run: npx playwright merge-reports --reporter html ./blobs
- name: Merge into JSON Report
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: /tmp/playwright-results.json
run: npx playwright merge-reports --reporter=json ./blobs
- name: Bench report
run: |
docker run --rm \
--volume="/tmp/playwright-results.json:/home/bench/tests/playwright-results.json" \
us-docker.pkg.dev/grafanalabs-global/docker-grafana-bench-prod/grafana-bench:v0.5.1 report \
--grafana-url "http://localhost:3000" \
--grafana-version "CI- ${{ github.sha }}" \
--test-suite-name "FrontendCore" \
--report-input playwright \
--report-output log \
--log-level DEBUG \
/home/bench/tests/playwright-results.json
- name: Upload HTML report
id: upload-html
uses: actions/upload-artifact@v5
uses: actions/upload-artifact@v7
with:
name: playwright-html-${{ github.run_number }}
name: playwright-html
path: playwright-report
retention-days: 7
- name: Check test suites
id: check-jobs
uses: ./.github/actions/check-jobs
uses: grafana/grafana/.github/actions/check-jobs@main
continue-on-error: true # Failure will be reported on Show test results step
with:
needs: ${{ toJson(needs) }}
@@ -503,15 +330,58 @@ jobs:
env:
FAILED: ${{ steps.check-jobs.outputs.any-failed }}
REPORT_URL: ${{ steps.upload-html.outputs.artifact-url }}
RUN_ID: ${{ github.run_id }}
# sed removes the leading `../../src/` from the paths
run: |
npx playwright merge-reports --reporter list ./blobs | sed 's|\(\.\./\)\{1,\}src/|/|g'
echo ""
echo "Run the following command to view the results locally:"
echo " ./scripts/view-playwright-ci.sh $RUN_ID"
echo ""
echo "Alternatively, download the test report from $REPORT_URL"
echo "and view it locally with:"
echo " yarn playwright show-report <path-to-unzipped-report>"
if [ "$FAILED" = "true" ]; then
echo ""
echo "Download the test report from $REPORT_URL"
exit 1
fi
report-playwright-bench:
needs:
- run-playwright-tests
if: ${{ !cancelled() }}
name: Report Playwright benchmarks
runs-on: ubuntu-x64-small
permissions:
contents: read
id-token: write
steps:
- name: Download blob reports from GitHub Actions Artifacts
uses: actions/download-artifact@v6
with:
path: blobs
pattern: playwright-blob-*
merge-multiple: true
- name: Merge into JSON Report
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: /tmp/playwright-results.json
run: npx playwright merge-reports --reporter=json ./blobs
- name: Bench report
run: |
docker run --rm \
--volume="/tmp/playwright-results.json:/home/bench/tests/playwright-results.json" \
us-docker.pkg.dev/grafanalabs-global/docker-grafana-bench-prod/grafana-bench:v1.0.1 report \
--service grafana \
--service-url "http://localhost:3000" \
--service-version "CI-${{ github.sha }}" \
--suite-name "FrontendCore" \
--run-stage ci \
--report-input playwright \
--report-output log \
--log-level DEBUG \
/home/bench/tests/playwright-results.json
publish-metrics:
name: Publish metrics
# Run on `grafana/grafana` main branch only
@@ -519,7 +389,7 @@ jobs:
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
runs-on: ubuntu-x64-small
steps:
- id: vault-secrets
uses: grafana/shared-workflows/actions/get-vault-secrets@main
@@ -527,44 +397,29 @@ jobs:
repo_secrets: |
GRAFANA_MISC_STATS_API_KEY=grafana-misc-stats:api_key
- name: Checkout code
uses: actions/checkout@v5
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@v6
uses: ./.github/actions/setup-node
with:
node-version-file: '.nvmrc'
- name: Install dependencies
run: yarn install --immutable
cache: 'false' # Don't use setup-node's built in caching, we'll handle it in the yarn-install step
- name: Install yarn dependencies
uses: ./.github/actions/yarn-install
- name: Extract and publish metrics
run: ./scripts/ci-frontend-metrics.sh | node --experimental-strip-types .github/workflows/scripts/publish-frontend-metrics.mts
env:
GRAFANA_MISC_STATS_API_KEY: ${{ env.GRAFANA_MISC_STATS_API_KEY}}
# This is the job that is actually required by rulesets.
# We want to only require one job instead of all the individual tests.
# Future work also allows us to start skipping some tests based on changed files.
# 'All E2E tests complete' is still a required check. To help with this transition, we keep
# a placeholder job to avoid needing to remove the required check from github yet.
required-e2e-tests:
needs:
- run-e2e-tests
- build-grafana
# a11y test is not listed on purpose: it is not an important E2E test.
# It is also totally fine to fail right now.
# always() is the best function here.
# success() || failure() will skip this function if any need is also skipped.
# That means conditional test suites will fail the entire requirement check.
- run-playwright-tests
if: always()
name: All E2E tests complete
runs-on: ubuntu-latest
runs-on: ubuntu-x64-small
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Check test suites
uses: ./.github/actions/check-jobs
with:
needs: ${{ toJson(needs) }}
failure-message: "One or more E2E test suites have failed"
success-message: "All E2E test suites completed successfully"
- name: Placeholder step
run: echo "This step is intentionally left blank. Check playwright-required-tests job for the actual test results."

2
e2e-playwright/dashboards-suite/dashboard-timepicker.spec.ts
View File

@@ -44,11 +44,13 @@ test.describe(
// Set user preferences for timezone
await page.goto('/profile');
await page.getByTestId(selectors.components.TimeZonePicker.containerV2).click();
await page.keyboard.type('Tokyo');
await page.getByRole('option', { name: 'Asia/Tokyo' }).click();
await page.getByTestId(selectors.components.UserProfile.preferencesSaveButton).click();
// wait for the page to reload before trying to navigate, otherwise this can cause flakes
// see e.g. https://github.com/microsoft/playwright/issues/21451#issuecomment-1502251404
await page.waitForURL('/profile');
await page.waitForLoadState('networkidle');
// Open dashboard with time range from 8th to end of 10th.
// Will be Tokyo time because of above preference

5
playwright.config.ts
View File

@@ -23,9 +23,8 @@ export function withAuth(project: Project): Project {
export const baseConfig: PlaywrightTestConfig<PluginOptions, {}> = {
fullyParallel: true,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
retries: process.env.CI ? 1 : 0,
workers: process.env.CI ? 4 : undefined,
reporter: [
['html'], // pretty
['./e2e-playwright/utils/axe-a11y/reporter.ts'], // accessibility reporter

23
scripts/view-playwright-ci.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -euo pipefail
RUN_NUMBER="${1:-}"
if [ -z "$RUN_NUMBER" ]; then
BRANCH="$(git rev-parse --abbrev-ref HEAD)"
echo "No run ID provided, looking up latest run for branch '$BRANCH'..."
RUN_NUMBER="$(gh run list --workflow=pr-e2e-tests.yml --branch="$BRANCH" --limit=1 --json databaseId --jq '.[0].databaseId')"
if [ -z "$RUN_NUMBER" ]; then
echo "Error: No runs found for branch '$BRANCH'"
exit 1
fi
echo "Using run $RUN_NUMBER"
fi
DEST="/tmp/playwright-html-${RUN_NUMBER}"
echo "Downloading artifact playwright-html from run $RUN_NUMBER"
gh run download "$RUN_NUMBER" --name playwright-html --dir "$DEST"
echo "Opening report..."
yarn playwright show-report "$DEST"

2
scripts/webpack/webpack.prod.js
View File

@@ -30,7 +30,7 @@ const envConfig = getEnvConfig();
module.exports = (env = {}) =>
merge(common(env), {
mode: 'production',
devtool: 'source-map',
devtool: process.env.NO_SOURCEMAP === '1' ? false : 'source-map',
entry: {
dark: './public/sass/grafana.dark.scss',