Files
grafana/scripts/drone/utils/utils.star
Jack Baldry 8379a5338c CI: Lint starlark files with buildifier (#59157)
* Add verify-starlark build action that returns an error for starlark files with lint

Relies on `buildifier` tool.

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Add verify_starlark_step to PR pipeline

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Manually fetch buildifier in curl_image until a new build_image is created

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Format with buildifier

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Remove all unused variables retaining one unused function

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Use snake_case for variable

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Replace deprecated dictionary concatenation with .update() method

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Start adding docstrings for all modules and functions

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Prefer os.WriteFile as ioutil.WriteFile has been deprecated since go 1.16

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Attempt to document the behavior of the init_enterprise_step

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document test_backend pipeline

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document enterprise_downstream_step

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document the pipeline utility function

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document publish_images_step

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document publish_images_steps

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document enterprise2_pipelines function

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Add tags table for Starlark files.

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document test_frontend

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document windows function

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Add docstrings to verifystarlark functions

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Refactor error handling to be more clear and document complex behavior

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Split errors into execution errors and verification errors

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document all other library functions

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Add local variables to TAGS

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Add blank line between all Args and Returns sections

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Fix new linting errors

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Lint new Starlark files

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Correct buildifier binary mv

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Document the need to set nofile ulimit to at least 2048

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Update build-container to include buildifier

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Ensure buildifier binary is executable

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Fix valid content test

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Simply return execution error

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Only check files rather than fixing them

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Use updated build-container with executable buildifier

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Test that context cancellation stops execution

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Simplify error handling

Return execution errors that short circuit WalkDir rather than
separately tracking that error.

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Remove fetching of buildifier binary now that it is in the build-container

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Use build image in verify-starlark step

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Use semver tag

The image is the same but uses a semver tag to make it clearer that
this is a forward upgrade from the old version.

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Use node 18 image with buildifier

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

---------

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
2023-01-30 09:27:11 +00:00

137 lines
4.3 KiB
Plaintext

"""
This module contains utility functions for generating Drone pipelines.
"""
load(
"scripts/drone/steps/lib.star",
"slack_step",
)
load("scripts/drone/vault.star", "pull_secret")
failure_template = "Build {{build.number}} failed for commit: <https://github.com/{{repo.owner}}/{{repo.name}}/commit/{{build.commit}}|{{ truncate build.commit 8 }}>: {{build.link}}\nBranch: <https://github.com/{{ repo.owner }}/{{ repo.name }}/commits/{{ build.branch }}|{{ build.branch }}>\nAuthor: {{build.author}}"
drone_change_template = "`.drone.yml` and `starlark` files have been changed on the OSS repo, by: {{build.author}}. \nBranch: <https://github.com/{{ repo.owner }}/{{ repo.name }}/commits/{{ build.branch }}|{{ build.branch }}>\nCommit hash: <https://github.com/{{repo.owner}}/{{repo.name}}/commit/{{build.commit}}|{{ truncate build.commit 8 }}>"
def pipeline(
name,
edition,
trigger,
steps,
services = [],
platform = "linux",
depends_on = [],
environment = None,
volumes = []):
"""Generate a Drone Docker pipeline with commonly used values.
In addition to the parameters provided, it configures:
- the use of an image pull secret
- a retry count for cloning
- a volume 'docker' that can be used to access the Docker socket
Args:
name: controls the pipeline name.
edition: used to differentiate the pipeline for enterprise builds.
trigger: a Drone trigger for the pipeline.
steps: the Drone steps for the pipeline.
services: auxilliary services used during the pipeline.
Defaults to [].
platform: abstracts platform specific configuration primarily for different Drone behavior on Windows.
Defaults to 'linux'.
depends_on: list of pipelines that must have succeeded before this pipeline can start.
Defaults to [].
environment: environment variables passed through to pipeline steps.
Defaults to None.
volumes: additional volumes available to be mounted by pipeline steps.
Defaults to [].
Returns:
Drone pipeline
"""
if platform != "windows":
platform_conf = {
"platform": {"os": "linux", "arch": "amd64"},
# A shared cache is used on the host
# To avoid issues with parallel builds, we run this repo on single build agents
"node": {"type": "no-parallel"},
}
else:
platform_conf = {
"platform": {
"os": "windows",
"arch": "amd64",
"version": "1809",
},
}
pipeline = {
"kind": "pipeline",
"type": "docker",
"name": name,
"trigger": trigger,
"services": services,
"steps": steps,
"clone": {
"retries": 3,
},
"volumes": [
{
"name": "docker",
"host": {
"path": "/var/run/docker.sock",
},
},
],
"depends_on": depends_on,
"image_pull_secrets": [pull_secret],
}
if environment:
pipeline.update(
{
"environment": environment,
},
)
pipeline["volumes"].extend(volumes)
pipeline.update(platform_conf)
if edition in ("enterprise", "enterprise2"):
# We have a custom clone step for enterprise
pipeline["clone"] = {
"disable": True,
}
return pipeline
def notify_pipeline(
name,
slack_channel,
trigger,
depends_on = [],
template = None,
secret = None):
trigger = dict(trigger)
return {
"kind": "pipeline",
"type": "docker",
"platform": {
"os": "linux",
"arch": "amd64",
},
"name": name,
"trigger": trigger,
"steps": [
slack_step(slack_channel, template, secret),
],
"clone": {
"retries": 3,
},
"depends_on": depends_on,
}
# TODO: this overrides any existing dependencies because we're following the existing logic
# it should append to any existing dependencies
def with_deps(steps, deps = []):
for step in steps:
step["depends_on"] = deps
return steps