* Get ResourceStats before indexing
* Replaced localcache.CacheService to handle expiration faster (localcache.CacheService / gocache.Cache only expires values at specific interval, but we need to close index faster)
* singleflight getOrBuildIndex for the same key
* expire only in-memory indexes
* file-based indexes have new name on each rebuild
* Sanitize file path segments, verify that generated path is within the root dir.
* Add comment and test for cleanOldIndexes.
* PoC: check if testing.Short is called from integration tests only.
* Rename helper function.
* Fix logic.
* Remove skipping of integration tests from non-integration tests.
* Remove skipping of integration tests from non-integration tests.
* Fix import.
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold
* Refactor mockResourceIndex
* Test setting of indexing threshold configs
* Tweak comments, log
* Fix logging in buildEmptyIndex
* Export and reuse TestDocumentBuilderSupplier
* Reuse MockResourceIndex
* Use authlib repo. Use otel
* Use interceptors on the provider level
* Create a new wire set with otel
* Lint
* Fix test
* make update-workflow
* make update-workspace
* make update-workspace. Try to add authlib as enterprise imports
* make update-workspace
* CI: Allow Bench conversion to fail
We shouldn't mark PRs and commits as X if they fail to convert logs with Bench.
* Fix: Prints should always include new lines
* fix: remove unused import
* Fix CodeQL warnings
* Also validate if path is a file above the safe dir
* Lint
* [REVIEW] reduce possibility of exploits
* Comment
* Remove test scenario
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric
* use wire to inject bleve index metrics
* remove sprinkles metrics from bleve index metrics
* log error when trying to calculate file index size and bump interval to 1m instead of 5s
* Move dashboard k8s APIs to a separate app
Signed-off-by: Igor Suleymanov <igor.suleymanov@grafana.com>
* Copy dashboard code in Dockerfile
Signed-off-by: Igor Suleymanov <igor.suleymanov@grafana.com>
* Fix conversion generation
Signed-off-by: Igor Suleymanov <igor.suleymanov@grafana.com>
* Update OpenAPI snapshot for dashboard/v0alpha1
Signed-off-by: Igor Suleymanov <igor.suleymanov@grafana.com>
---------
Signed-off-by: Igor Suleymanov <igor.suleymanov@grafana.com>
* WIP adding custom analyzer so we can do substring search efficiently
* Adding unit tests for title search
* formatting
* adds more title search unit tests
* organize helpers
* fixes issue caused by having two title mappings
* Removes camelcase token filter since it prevents you from searching for a substring of chars and numbers. Adds regression test.
* adds back mapping for title_phrase
* use simple analyzer for input query string so it doesn't filter out english stop words
* ran bleve tests, table snapshots updated
* ignore linter for "unused" test functions. They are very helpful for troubleshooting search. Keeping them.
* only log total hits and query cost if result not nil
* fixes failing test - one more field because there are two title mappings now
* fix test
* fixes test - only take first item when its the title
* Adds separate internal field for title ngram mapping.
When searching with a query, results are sorted by score desc.
When searching without a query, results are sorted by title desc.
Adjusts ngram max to be 10.
Text queries are a disjunction of an exact match, phrase match, and a match. Boosted to have priority in that order.
Adds more unit tests for searching.
* linter
* fix test
* ran tests - generated new test dash json
* sort by title phrase instead of title
* fix test - not relying on /apis/dashboard/search to apply title sorting anymore
* wires up dashboards page to be able to sort by usage stats (sprinkles)
* dont mutate field
* use better type for field
* adds tests. Had to export some types and put the field type back to object.
* frontend asks for sort field in response if needed
* adds some unit tests for getSortOptions
* use Record instead of object
* prettier
* adds ternaries, another unit test
* add feature flag
* puts search permission filtering behind a feature flag
* fixes issue with doc match id. When the match is for an in-memory index, the internal id is a string (this is what we expected). However, when its a file-based index, the internal id is a binary encoded int64 that point to something internally. So to get the id, we need to use ExternalID() instead of relying on the indexInternalID to be the correct format.
* adds debug log
* update comment
* formatting
* fix bug when parsing results in search handler
* applies permissions filtering to bleve query
* formatting
* wraps in check for access being present, adds some comments
* update go mod
* fix tests
* add dep owner
* fix go mod
* add space after //
* clean up returns
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
* fixed formatting
* Uses single checker since index is for single resource. Passes folderId using dvReader to checker func. Adds debug logging.
* handles federation with index permission checkers
* formatting
* move import
---------
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
* wip. adding sprinkles fields.
* some refactoring. Works with sprinkles now.
* exclude top level dashboard hit fields from hit "fields"
* adds unit test for DecodeCell helper
* test can search for specific dashboard fields on bleve index
* adds search handler tests for the fields and tests for fields when transforming the search req to a bleve search req
* fix panic when calling fields.Set() with int32
* adds regression test
* remove unneeded method on test mock client
* fix linter issues
* updates dashboard test data for bleve tests
* remove DASHBOARD_LEGACY_ID from bleve_tests
* dont cast twice
* updates test to sort by dashboard_views_last_1_days
* declare excludedFields outside of function
* fixes sorting by dashboard fields - prepends "fields." to any dashboard fields we try to sort by
* uses map for excludedFields
* expects fields to be array-style url param
* change method name
* fixes failing tests - needed to add column type to mocks
