* adding the additional datanode start step
* switch to correct guard in remote reindexing
* fixing test
* moving steps a bit more, fixing tests
* changing startup
* changing startup, fixing states etc.
* fix test
* fix: do not set preflight result to FINISHED for reindexing branch
---------
Co-authored-by: Matthias Oesterheld <matthias.oesterheld@graylog.com>
* Use official mongojack dependency
* Use unset instead of set(field, null) to prevent serialization errors
* Use different wrappers for single/multi update results
* Fix id conversion for ObjectId
* Remove unused method
* Always return saved object
* Handle null id in conversion
* Remove @Id annotation from constructor
* Extract saved ID via document codec
* Fix implementation of getN()
* Fix implementation of isUpdateOfExisting
* Add "sparse" index option
* Don't rely on in-place updates
* Fix #distinct
* Prevent ID injection into value classes
* Use ObjectIDs in filter queries matching "_id" fields
* Add package-info.java
* Add support for BsonTimestamp to Joda DateTime conversion
* Replace deprecated classes with Bson in method signatures
* Add tests that pass with both old and new implementation
* Deprecate CommonMongoJackObjectMapperProvider
* Add changelog
* Add upgrade note
* Remove handling of removed data type
* Refresh underlying cursor when find options are set
* post configure event to start Jersey server
* fallback to any node if no leader node present
* remove duplicate event post for secure setups, remove duplicate code
Add a new safe_classes configuration option to restrict the classes allowed to be used
as cluster config and event types.
The configuration option allows to specify a comma-separated set of prefixes matched
against the fully qualified class name.
For now, the default value for the configuration is org.graylog.,org.graylog2., which will
allow all classes that Graylog maintains.
This should work out of the box for almost all setups. Changing the default value might
only be necessary if external plugins require cluster config or event types outside the
"org.graylog." or "org.graylog2." namespaces. If that is the case, the configuration setting
can be adjusted to cover this use case, e.b. by setting it to
safe_classes = org.graylog.,org.graylog2.,custom.plugin.namespace.
if said classes are located within the custom.plugin.namespace package.
Refs: https://github.com/Graylog2/graylog2-server/security/advisories/GHSA-p6gg-5hf4-4rgj
* added new states
* fix error message
* adding new state to test
* removed doubled line
* start if insecure has been chosen
* adding jump
* always start if everything's confgured
* fixing test
* guide, first version
* fix for correct start
* fixed typo
* adding script to create local docker images
* adding a little more info regarding locally generated docker images
* added macOS info
* added OpenSearch 1.3 info
* added Elasticsearch 7.10 info
* Update Migration.md
* renamed migration to manual to make scope clear
* added changelog
* State machine and rest api for datanode migration
* added changelog
* added license
* renamed remote reindex resource api
* fixed RemoteReindexingMigrationIT
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* Adding new additional dependencies.
* Setting up injection infrastructure for new client in parallel.
* Adding license headers.
* Including new deps in test jar.
* Removing jakarta deps from assembly.
* Building separate jars for integration tests & distribution.
* Fix class loader issue
Exclude "module-info.class" files from the graylog-storage-opensearch
JAR file to avoid class loader issues.
Revert previous assembly and dependency changes that worked around the
issue in tests.
* Adjusting to Jakarta and recent data node changes.
* fixing wrong builder usage
---------
Co-authored-by: Bernd Ahlers <bernd@graylog.com>
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* Create Data Stream template, prepare ism policy
* add ism
* create ism and collect jvm heap as first metric
* change lookup and term query to use data streams
* start moving data stream functionality to graylog
* move data stream management into graylog
* code cleanup, add tests
* better validation, clean up after tests
* additional clean up after IT
* don't remove `index-management` plugin from datanode
* inject ObjectMapper
* change number of replicas from default 1 to 0 for test
* set `number_of_replicas` to 0 for .opendistro-ism-config in test
* replace javax with jakarta classes
* fix integration test
* remove data streams on test cleanup
* set high priority for data stream template
* Update to OpenSearch 2.11.0
* Update to OpenSearch 2.11.0
* reverting test back
* reverting test back
* better handling of waiting for datanode startups in integration tests
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
* ensure password_secret meets length requirement
* update CL
* update test
* adjusted wording so it references the data node
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* Parser for opensearch/elasticsearch data directories
* add opensearch 1.3 and elasticsearch 6.8.0 to indexer parser
* fixed paths
* code cleanup
* remove explicit dependency version of jackson smile
* Preflight check for index dir compatibility
* indexer dir parser fixed for empty dirs
* added changelog
* rest controller for indices directory parser
* require auth for indices compatibility rest api
* catch incompatibility exceptions of indices dir parsing
* code cleanup
* better dir detection handling
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
- Read the "max_map_count" from the /proc system instead of calling the
/sbin/sysctl command
- Only run check on Linux operating systems (avoids warning on FreeBSD)
- Throw "PreflightCheckException" instead of "RuntimeException"
- Don't use "System.out.println()" for logging
- Enhance changelog entry to help users to understand the fix
Refs https://github.com/Graylog2/graylog2-server/pull/17436
* publish data node status
* add paginated datanode service
* changelog
* adjust changelog
* adjust NodeImplTest
* refactor initial separation of server node and data node
* fix backward compatibility
* refactor separation of server node and data node
* add timezone
* add new service tests
* add entity tests
* inject DataNodeDto NodeService
* service, rest & permissions for data node management
* add tests
* add reset data node event
* add states and transitions for node removal
add node removal functionality
* changelog
* add datanode permissions to permission list
* add audit event types to list
* add custom thread factory
* add node removal integration test
refactor rest calls in ITs
* wait for primary shard node availability
* refactor duplicate code
* simplify node retrieval
* add `@OnlyInSecuredNode` annotation and request filter
move delete endpoint to management controller and secure it
adjust IT
change javadoc
* stabilize tests
The caching in the latest upstream version is completely broken since
April 2023 and the maintainers are currently unresponsive.
Our fork is based on version 1.6.8 and fixes the excessive download
progress logging which is a problem with the log size limit on
GitHub Actions.
* using the hostname as the default node-name, if none was specified explicitly
* using the hostname as the default node-name, if none was specified explicitly
* using the hostname as the default node-name, if none was specified explicitly
* nodeName is either set explicitly in the conf file or will be the result of the hostname setting otherwise
* small refactoring
* Fix file permissions for key material #17009
* added changelog
* correct file permissions for private key used during preflight CSR
* better http and transport keystore files generation
* removed changelog
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
