This periodically logs how many times Resource Manager limits were
exceeded. If they aren't exceeded, then nothing is logged. The log
levels are at ERROR log level so that they are shown by default.
The motivation is so that users know when they have exceeded resource
manager limits. To find what is exceeding the limits, they'll need to
turn on debug logging and inspect the errors being logged. This could
collect the specific limits being reached, but that's more complicated
to implement and could result in much longer log messages.
* feat: persist limit changes to config
This changes the "ipfs swarm limit" command so that when limit changes
are applied via the command line, they are persisted to the repo
config, so that they remain in effect when the daemon restarts.
Any existing limit.json can be dropped into the IPFS config easily
using something like:
cat ~/.ipfs/config | jq ".Swarm.ResourceMgr.Limits = $(cat limit.json)" | sponge ~/.ipfs/config
This also upgrades to Resource Manager v0.3.0, which exports the config
schema so that we don't have to maintain our own copy of it.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* update go-libp2p to v0.18.0
* initialize the resource manager
* add resource manager stats/limit commands
* load limit file when building resource manager
* log absent limit file
* write rcmgr to file when IPFS_DEBUG_RCMGR is set
* fix: mark swarm limit|stats as experimental
* feat(cfg): opt-in Swarm.ResourceMgr
This ensures we can safely test the resource manager without impacting
default behavior.
- Resource manager is disabled by default
- Default for Swarm.ResourceMgr.Enabled is false for now
- Swarm.ResourceMgr.Limits allows user to tweak limits per specific
scope in a way that is persisted across restarts
- 'ipfs swarm limit system' outputs human-readable json
- 'ipfs swarm limit system new-limits.json' sets new runtime limits
(but does not change Swarm.ResourceMgr.Limits in the config)
Conventions to make libp2p devs life easier:
- 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager
- 'limit.json' overrides implicit defaults from libp2p (if present)
* docs(config): small tweaks
* fix: skip libp2p.ResourceManager if disabled
This ensures 'ipfs swarm limit|stats' work only when enabled.
* fix: use NullResourceManager when disabled
This reverts commit b19f7c9eca4cee4187f8cba3389dc2c930258512.
after clarification feedback from
https://github.com/ipfs/go-ipfs/pull/8680#discussion_r841680182
* style: rename IPFS_RCMGR to LIBP2P_RCMGR
preexisting libp2p toggles use LIBP2P_ prefix
* test: Swarm.ResourceMgr
* fix: location of opt-in limit.json and rcmgr.json.gz
Places these files inside of IPFS_PATH
* Update docs/config.md
* feat: expose rcmgr metrics when enabled (#8785)
* add metrics for the resource manager
* export protocol and service name in Prometheus metrics
* fix: expose rcmgr metrics only when enabled
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: rcmgr_metrics.go
* refactor: rcmgr_defaults.go
This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled
We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p
decides to change defaults in any of the future releases.
* refactor: adjustedDefaultLimits
Cleans up the way we initialize defaults and adds a fix for case
when connection manager runs with high limits.
It also hides `Swarm.ResourceMgr.Limits` until we have a better
understanding what syntax makes sense.
* chore: cleanup after a review
* fix: restore go-ipld-prime v0.14.2
* fix: restore go-ds-flatfs v0.5.1
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* multibase encoding on pubsub
* emit multibase for json clients
* refactor(pubsub): base64url for all URL args
This makes it easier to reason about.
Also added better helptext to each command explaining how the binary
data is encoded on the wire, and how to process it in userland.
* refactor: remove ndpayload and lenpayload
Those output formats are undocumented and seem to be only used in tests.
This change removes their implementation and replaces it with error
message to use JSON instead.
I also refactored tests to test the --enc=json response format instead
of imaginary one, making tests more useful as they also act as
regression tests for HTTP RPC.
* test(pubsub): go-ipfs-api
Testing against compatible version from
https://github.com/ipfs/go-ipfs-api/pull/255
* refactor: safeTextListEncoder
Making it clear what it does and why
* refactor(pubsub): unify peerids
This ensures `ipfs pubsub sub` returns the same peerids in the `From`
field as `ipfs pubsub peers`.
libp2p already uses base encoding, no need to double wrap or use custom
multibase.
* test(pubsub): go-ipfs-http-client
* refactor(pubsub): make pub command read from a file
We want to send payload in the body as multipart so users can use
existing tools like curl for publishing arbitrary bytes to a topic.
StringArg was created for "one message per line" use case, and if data
has `\n` or `\r\n` byte sequences, it will cause payload to be split. It
is not possible to undo this, because mentioned sequences are lost, so
we are not able to tell if it was `\n` or `\r\n`
We already avoid this problem in `block put` and `dht put` by reading
payload via FileArg which does not mangle binary data and send it as-is.
It feel like `pubsub pub` should be using it in the first place anyway,
so this commit replaces StringArg with FileArg.
This also closes https://github.com/ipfs/go-ipfs/issues/8454
and makes rpc in go-ipfs easier to code against.
* test(pubsub): publishing with line breaks
Making sure we don't see regressions in the future.
Ref. https://github.com/ipfs/go-ipfs/issues/7939
* chore: disable pubsub interop for now
See
344f692d8c
* test: t0322-pubsub-http-rpc.sh
- Adds HTTP RPC regression test that ensures topic is encoded as URL-safe
multibase.
- Moves pubsub tests to live in unique range ./t032x
* fix(ci): js-ipfs with fixed pubsub wire format
uses js-ipfs from https://github.com/ipfs/js-ipfs/pull/3922
until js-ipfs release can ship with dependency on go-ipfs 0.11.0-rc1
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Most of these are probably harmless but a few looked like they might actually be
bugs. Most of them are just faulty tests.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
* Updates go-ipfs-cmds to try to get the tests to pass on travis.
* While we're at it, fix duplicate gx deps.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
Switch _back_ to the 0.4.18 style of peer IDs while we figure things out. See
https://github.com/libp2p/specs/issues/138.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
