47 Commits

Author SHA1 Message Date
3fad6183aa CloudMigrations: Create resource dependency map to validate snapshot request (#102594)
* CloudMigrations: Create resource dependency map to validate snapshot request

* CloudMigrations: Validate resource types dependencies in create snapshot request

* CloudMigrations: Update service interface to pass parsed resource types for creation

* CloudMigrations: Conditionally append resource to snapshot if enabled

* CloudMigrations: Add /cloudmigration/resources/dependencies endpoint

* CloudMigrations: Properly filter dashboards and folders from snapshot
2025-04-08 17:46:25 +02:00
88e51d549c CloudMigrations: Add sorting and error filtering to Snapshot Results backend (#102753)
* implement sorting

* swagger gen

* minor fixes

* clean up param reading

* add todo

* add errors only prop

* codegen stuff

* fix copy paste error

* forgot the api gen

* cleanup

* remove tests that are obe

* fix test
2025-03-30 10:42:45 -04:00
fde475e3d9 CloudMigrations: save snapshot of alert rule groups (#100109) 2025-02-06 12:35:59 +01:00
c9d22f06c3 CloudMigrations: Bulk update local resources (#96002)
* wip

* make tests pass

* get all tests passing

* fixes

* some small cleanup

* fix test

* convert delimiter keys to struct keys

* dont execute empty sql statement

* remove printlns

* fix unit test

* a bit more cleanup

* whoops
2025-01-10 14:42:18 -05:00
1699dfa307 CloudMigrations: Add support for migration of plugin resources (#95612)
* start plugins migration

* more plugin work

* add warning

* fakepluginsettings test

* tests get plugins

* lint

* load logos

* go lint

* get all plugins once

* locales

* josh suggestion to inject query in rtk

* more plugin filters

* remove datasource warning

* access control for plugins

* remove unused method

* lint

* use gcom list
2025-01-06 10:37:15 +00:00
750a0bed71 MigrationAssistant: Restrict dashboards, folders and datasources by the org id of the signed in user (#96339)
apply security patch: main/206-202410241510.patch
2024-11-12 13:03:16 -06:00
69f185b459 CloudMigrations: handle provisioned data source error (#95473) 2024-10-28 12:02:46 +01:00
152b83b3fb CloudMigrations: handle more error codes from GMS backend (#95257)
* CloudMigrations: remove unnecessary if feature toggle gating

* CloudMigrations: handle new error codes from backend
2024-10-24 15:55:02 +02:00
98e5048370 CloudMigration - Display different error messages for create migration errors (#94683)
* start on tokens

* more error messages

* more handling

* rephrased with suggestions from Daniel

* separate gms parse method

* use translation

* refactor initial idea to use error obj

* use error dto result

* handle gms client

* clean logs and comments

* fix tests

* tests for gms

* test and lint

* lint

* one more handling from gms

* typing in fe

* use error interface

* use validation error

* remove unused gms error

* use errorlib and helper function in fe

* regen api

* use same error util

* one more error to handle
2024-10-21 09:45:54 +01:00
a50507e645 CloudMigration: Interpret error code from migration resource item (#94407)
* start on loading the error code

* error code to message mapping

* use resource code type

* use defined error code

* partial updates from comments

* i18nKey gen

* fixed t

* fixed translations

* typing
2024-10-17 14:09:09 +01:00
9af095d730 CloudMigrations: Add data types for alerts resources (#94251) 2024-10-07 12:52:31 +02:00
d88be2819d CloudMigrations: Store parent folder name in cloud_migration_resource table (#94009)
* use name in fe

* store parent folder name in local db

* clean up

* tiny test

* trial react

* rename to parent name

* go lint

* generate api and ts

* go tests

* rearrange

* clean

* update with suggestions from josh

* make library elements work

* updates from comments

* global migration types

* parent name for alter table
2024-10-07 11:35:08 +01:00
1a31abe254 CloudMigrations: Limit frontend query to get latest snapshots (#93639)
* latest param to endpoint and adapt frontend query

* change to sort param

* api

* remove description
2024-10-01 09:28:25 +01:00
1b7cd4c684 Cloud migrations: store resource name in the cloud_migration_resource table (#93612)
* Cloud migrations: store resource name in the cloud_migration_resource table

* remove unused function: convertMigrateDataResponseToDTO

* make swagger-clean && make openapi3-gen

* use DB_Text for cloud_migration_resource.name instead of DB_Varchar
2024-09-26 11:16:52 -03:00
14abf2dab3 CloudMigrations: add Library Element as a valid migration resource type (#93781) 2024-09-26 15:29:28 +02:00
6cd0971dc6 CloudMigration: Show warning message for successfully migrated non-core data sources (#91545)
* minor performance improvement

* apply a warning to any non-core plugins that successfully migrate

* commit frontend wip while I refactor some stuff

* update api

* repurpose error dialog to be a generic details dialog

* whitespace

* add unit test

* fixes from testing

* fix migration summary

* add comment

* fix localization stuff

* fix backend test

* reduce number of queries to the db

* some PR feedback

* whitespace
2024-08-06 19:35:10 +03:00
3e138449bb CloudMigrations: Fix bugs found during local testing (#91163)
* send dashboard commands instead of dashboards

* move status updates before goroutine to ensure frontend polls

* fix syncing issues between snapshot state and resources

* make sessionUid a requirement for modifying snapshots

* move the function I meant to move earlier

* remove accidental commit

* another accidental commit

* verify UpdateSnapshot is called with sessionUid

* revert

* pass in session uid everywhere

* forgot to save

* fix unit test

* fix typo

* tiny tweak
2024-07-30 09:02:41 -04:00
dc355331a6 CloudMigrations: Store encryption key in unified secrets table (#90908)
* store encryption key in unified secrets table

* fix local dev mode

* make metadata more realistic

* fix tests

* fix sql queries against postgres

* fix stats endpoint
2024-07-25 06:34:25 +03:00
8d8f2ba587 Cloud migration: encryption key is a byte slice (#90739)
* Cloud migrations: include metadata returned by CMS in the index file

* Cloud migrations: make EncryptionKey a []byte in structs

* test
2024-07-22 11:25:12 -03:00
07685646a3 Cloud migrations: include metadata returned by CMS in the index file (#90658) 2024-07-22 10:11:57 -03:00
9b7e9d992b CloudMigrations: Query GMS for a presigned upload url at upload time (#90505)
query GMS for an upload url at upload time
2024-07-17 11:53:21 -04:00
6ff21726b7 CloudMigrations: Query GMS for snapshot status with a results offset (#90453)
* query GMS for status with an offset

* remove unused state
2024-07-16 09:04:21 -04:00
542a1bf3ac CloudMigrations: Query Grafana Migration Status for status while the snapshot is in the cloud (#90314)
* implement querying gms for snapshot status

* add some documentation

* provide snapshot resources after snapshot is created

* add rate limiting to backend

* fix compilation error

* fix typo

* add unit tests

* finish merge

* lint

* swagger gen

* more testing

* remove duplicate test

* address a couple PR comments

* update switch statement to a map

* add timeouts to gms client through the http client

* remove extra whitespace

* put method back where it was so the PR is less confusing

* fix tests

* add todo

* fix final unit test
2024-07-15 09:22:57 -04:00
420ff1d07b Cloud migration: upload snapshot files using presigned url (#90273)
* Cloud migration: upload snapshot files using presigned url

* log error if index file cannot be closed

* log error if file cannot be closed in uploadUsingPresignedURL
2024-07-11 09:32:02 -03:00
317da43a84 CloudMigration: Provide a stats rollup in the GetSnapshot response (#90252)
* order session list descending

* add snapshot status method to store

* query stats while retrieving snapshot

* return stats in dto

* swagger

* fix tests

* commit results of bingo get

* fix swagger

* minor improvement

* fix typo

* forgot a file
2024-07-10 08:46:38 -04:00
d1952bb681 Cloud migrations: create snapshot files (#89693)
* Cloud migrations: create snapshot and store it on disk

* fix merge conflicts

* implement StartSnapshot for gms client

* pass snapshot directory as argument to snapshot builder

* ensure snapshot folder is set

* make swagger-gen

* remove Test_ExecuteAsyncWorkflow

* pass signed in user to buildSnapshot method / use github.com/grafana/grafana-cloud-migration-snapshot to create snapshot files

* fix FakeServiceImpl.CreateSnapshot

* remove new line
2024-07-03 10:38:26 -03:00
4d69213829 CloudMigrations: Break snapshot resources out into their own table (#89575)
* create a new table for migration resources

* remove raw result bytes from db

* more snapshot resource management stuff

* integrate new table with snapshots

* pass in result limit and offset as params

* combine create and update

* set up xorm store test

* add unit tests

* save some cpu

* remove unneeded arg

* regen swagger

* fix bug with result processing

* fix update create logic so that uid isn't required for lookup

* change offset to page

* regen swagger

* revert accidental changes to file

* curl command page should be 1 indexed
2024-06-24 23:50:07 -04:00
8a8f97b0e4 CloudMigrations: Implement snapshot management apis (#89296)
* add new apis

* add payloads

* create snapshot status type

* add some impl

* finish implementing update

* start implementing build snapshot func

* add more fake build logic

* add cancel endpoint. do some cleanup

* implement GetSnapshot

* implement upload snapshot

* merge onprem status with gms result

* get it working

* update comment

* rename list endpoint

* add query limit and offset

* add helper method to snapshot

* little bit of cleanup

* work on swagger annotations

* manual merge

* generate swagger specs

* clean up curl commands

* fix bugs found during final testing

* fix linter issue

* fix unit test
2024-06-19 09:20:52 -04:00
9d3a4e236d CloudMigrations: Refactor API for async work (#89084)
* rename some stuff

* more renaming

* clean up api

* rename more functions

* rename cms -> gms

* update comment

* update swagger gen

* update endpoints

* overzealous

* final touches

* dont modify existing migrations

* break structs into domain and dtos

* add some conversion funcs

* fix build

* update frontend

* try to make swagger happy
2024-06-13 17:58:59 +00:00
99d8025829 Chore: Move identity and errutil to apimachinery module (#89116) 2024-06-13 07:11:35 +03:00
83543c6b12 Cloud migrations: create route to fetch cloud migration token (#88176)
* Cloud migration: create route to fetch cloud migration token

* implement gcomStub.ListTokens

* fix swagger for POST /cloudmigration/migration

* fix swagger for POST /cloudmigration/migration

* fix swagger for POST /cloudmigration/migration
2024-05-31 09:39:10 -03:00
14ccf7f6d1 Cloud migrations: return errutil.Error in http responses (#87448) 2024-05-13 00:22:46 -04:00
6a3ab71d77 cms: clean up model file (#87179)
* move CreateAccessTokenResponseDTO to api.go

* remove unused types

* run make swagger-clean && make openapi3-gen
2024-05-13 00:22:19 -04:00
0719f73f35 Cloud Migration: Remove ID migration in favor of UID (#86550)
* - Added migration to add a new collumn UID to both migration and migration_run tables.
- Added migration to set UID for records already existent in the database before adding a new column.
- Added UID field to CloudMigration and CloudMigrationRun dtos (model.go)

* Fix db migration

* Updating store and model

* Updating API to use UID

* fix typo

* update openapi specs and generated endpoints

* fix spec

* update openapi specs and generated endpoints

* Fixing db mapping

* Fix frontend

* fix migration

* remove migration uid from run fetch endpoint

* Revert "merge"

This reverts commit d654e4c530caebe439060c876c44132e74a28de8, reversing
changes made to 5fe0b483eb62cc9dfceb01043d2165617d974ed9.

* manual merge

* rename some funcs for consistency

* make interfaces consistent

* validate uids

* update generated frontend api

* fix enterprise spec

* manually resolve api

* try again

* try yet again

* once more

---------

Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: joshhunt <josh@trtr.co>
2024-05-01 12:29:25 -04:00
7ce1254913 CloudMigration: Remove details from the migration run list (#87104)
Remove details from the migration run list
2024-04-30 11:44:35 -07:00
5ce8b60878 E2C: Resources table refactor (#85585)
* E2C: Resources table refactor

* update swagger spec with enums

* use native resource item type, rather than our mock type

* unit tests for resources table

* update spec
2024-04-05 10:08:42 +01:00
aad79c9400 E2C: Start connecting on-prem to real apis (#85527)
* E2C: Start connecting on-prem to real apis

* actually run the migration

* show resources

* basic dashboards resources

* show dashboard title

* remove console logs

* cleanup 1

* i18n

* disconnect

* i18n

* restore type

* use fixed format

* fix
2024-04-03 13:47:02 -04:00
a457ab3192 Cloudmigration: decode json data (#85548)
* decode get

* decode bytes

* response

* .

* linter

* quick fixes

---------

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
2024-04-03 18:47:49 +02:00
e08b5bd456 E2C: Fix Swagger spec for createMigration post body (#85525)
* E2C: Fix Swagger spec for CreateCloudMigration post body

* actually, don't rename that type
2024-04-03 11:59:50 +00:00
b885da09da CloudMigrations: Implement migrations API (#85348)
* Implement run migration endpoint

* Refactor RunMigration method into separate methods

* Save migration runs fix lint

* Minor changes

* Refactor how to use cms endpoint

* fix interface

* complete merge

* add individual items

* adds tracing to getMigration

* linter

* updated swagger definition with the latest changes

* CloudMigrations: Implement core API handlers for cloud migrations and migration runs (#85407)

* implement delete

* add auth token encryption

* implement token validation

* call token validation during migration creation

* implement get migration status

* implement list migration runs

* fix bug

* finish parse domain func

* fix urls

* fix typo

* fix encoding and decoding

* remove double decryption

* add missing slash

* fix id returned by create function

* inject missing services

* finish implementing (as far as I can tell right now) data migration and response handling

* comment out broken test, needs a rewrite

* add a few final touches

* get dashboard migration to work properly

* changed runMigration to a POST

* swagger

* swagger

* swagger

---------

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
Co-authored-by: Michael Mandrus <41969079+mmandrus@users.noreply.github.com>
2024-04-03 13:36:13 +02:00
4e89267603 Cloudmigration: swagger for the cloudmigration api (#85255)
* swagger for create token

* tried to fix errors

* update swagger annotations

* update swagger annotations

* update api GetMigrationList to return the *Response directly

* clean up previous commit

---------

Co-authored-by: joshhunt <josh@trtr.co>
2024-04-02 13:57:42 +02:00
383ebb2bc4 Cloudmigration: create migration (#85386)
* Cloudmigration: create migration

* fix drone build

* lint fix

* fix unit test

---------

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
2024-03-29 01:55:27 +01:00
5a5f76ae0a CloudMigrations: Add instance metadata to auth token (#85381)
* update how tokens are passed around

* rename structs
2024-03-28 15:02:54 -04:00
5b147d0847 CloudMigration: Add service to list all migrations (#85308) 2024-03-28 08:50:31 -03:00
a2e21eac8c Cloud migrations: create endpoint to create an access token (#84690)
* fix merge conflicts

* make token expiration configurable
2024-03-25 12:43:28 -03:00
e2f155f9f7 Grafana: define the api for the grafana cloudmigration api (not the csm api) (#84430)
* Add cloud migration endpoints

* Built auth into creating a migration.

* Added more detail to the migration result model

* goimports

* Update pkg/services/cloudmigration/api/api.go

Co-authored-by: lean.dev <34773040+leandro-deveikis@users.noreply.github.com>

* Update pkg/services/cloudmigration/api/api.go

Co-authored-by: lean.dev <34773040+leandro-deveikis@users.noreply.github.com>

---------

Co-authored-by: Leonard Gram <leo@xlson.com>
Co-authored-by: lean.dev <34773040+leandro-deveikis@users.noreply.github.com>
2024-03-25 13:30:47 +01:00
80447dd0cc Restructure cloudmigration service (#83211)
* Restructure cloudmigation service

* Adjust codewoners and wire

* Comment out unused metrics
2024-02-26 14:52:16 +01:00