* Initial pass at Telemetry plugin
Currently, IP Shipyard, with the help of Probelab, monitor and extract
Amino/IPFS public network metrics with the use of DHT crawlers and
bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and
their AgentVersion/Addresses obtained from the `identify` protocol, which
provides insights into protocol usage, total number of peers etc.
We would like to increase the ability to obtain more insights from the network
by collecting some more information in the future, but also to give users more
control over this collection (i.e. opt-out). The information collected will
not allow unique identification of anyone and is only used for aggregation.
Now, this PR explores a way of moving in this direction:
* A new "telemetry" fx plugin is in charge of dealing with telemetry
* The FX plugin allows to plug and make decisions / take actions during the setup phase:
* We can inspect whether we are using Private Networks before the libp2p.Host has been initialized.
* We can send telemetry after the libp2p Host is initialized.
* Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code.
As for behaviour:
* The user can opt-in/out via EnvVar, file in the repo path or plugin configuration.
* Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored.
* Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out.
The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer.
* checkpoint
* telemetry plugin: second pass
* On first run it generates a UUID and shows a message to the user.
* UUID is persistend to "telemetry_uuid"
* Sends telemetry 1 minute after boot and every 24h
* LogEvent is the thing containing all the telemetry that is sent
* Opt-out possible via env-var or plugin configuration
* Telemetry: add changelog and environment variable documentation
* docs: improved daemon message
making it more obvious nothing was sent yet
and that user had 15m to out-out
plus some debug logs that confirm opt-out
* refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY
* fix: add User-Agent header to telemetry requests
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* chore: update to go-log/v2
go-log v2 has been out for quite a while now and it is time to deprecate v1.
Replace all use of go-log with go-log/v2
Makes /api/v0/log/tail useful over HTTP
Updates dependencies that have moved to go-lov/v2
Removes support for ContextWithLoggable as this is not needed for tracing-like functionality
- Replaces: PR #8765
- Closes issue #8753
- Closes issue #9245
- Closes issue #10809
Other fixes:
* update go-ipfs-cmds
* update http logs test
* fix test
* Read/send one line of log data at a time
* Update -log-level docs
* Upgrade to pebble v2.0.3
- Configure latest pebble database format at init
- Do not automatically ratchet database format if set in config
- Daemon messge about new available pebble format
- Document pebble config with formatMajorVersion
- Add warning to users running badger, nudging them to switch to flatfs or pebble
- docs: explain Pebble's `FormatMajorVersion`
- Use pebbleds instead of badgerds in t0060-daemon.sh
- Print badgerds warning message to stderr
* Update pebble db to latest format by default
If the pebble database format is not explicitly set in the config, then set it to the latest format version by default. This will ensure that the database format is sufficiently up-to-date to be compatible with a major version upgrade of pebble.
* chore: stop using go-homedir
The `github.com/mitchellh/go-homedir` repo is archived, no longer needed, and no longer maintained.
- `homedir.Dir` is replaced by the stdlib `os.UserHomeDir`
- `homedir.Expand` is replaced by fsutil.ExpandHome` in the `github.com/ipfs/kubo/misc/fsutil` package.
Additional functionality, such as `DirWritable` and `FileExists` was moved into or included in the `github.com/ipfs/kubo/misc/fsutil` package.
* include pebble as built-in plugin
Pebble provides a high-performance alternative to leveldb as the datastore, and will serve as a replacement for badger1.
There are a number of tuning parameters available for tuning pebble's performance to your specific needs. Default values are used for any that are not configured or are set to the parameter's zero-value.
Requires https://github.com/ipfs/go-ds-pebble/pull/39Closes#10347
* docs: remove mention of ipfs-ds-convert. Rationale: https://github.com/ipfs/ipfs-ds-convert/issues/50
* docs: pebbleds profile
* test: meaningful t0025-datastores.sh
* Update config/init.go
* Update docs/config.md
* Do not hard-code zero values into pebble config
Fixes#8492
This introduces "nopfs" as a preloaded plugin into Kubo
with support for denylists from https://github.com/ipfs/specs/pull/383
It automatically makes Kubo watch *.deny files found in:
- /etc/ipfs/denylists
- $XDG_CONFIG_HOME/ipfs/denylists
- $IPFS_PATH/denylists
* test: Gateway.NoFetch and GatewayOverLibp2p
adds missing tests for "no fetch" gateways one can expose,
in both cases the offline mode is done by passing custom
blockservice/exchange into path resolver, which means
global path resolver that has nopfs intercept is not used,
and the content blocking does not happen on these gateways.
* fix: use offline path resolvers where appropriate
this fixes the problem described in
https://github.com/ipfs/kubo/pull/10161#issuecomment-1782175955
by adding explicit offline path resolvers that are backed
by offline exchange, and using them in NoFetch gateways
instead of the default online ones
---------
Co-authored-by: Henrique Dias <hacdias@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This adds a plugin interface that lets the plugin modify the fx
options that are passed to fx when the app is initialized. This means
plugins can inject their own implementations of IPFS interfaces. This
enables granular customization of go-ipfs behavior by plugins, such
as:
- Bitswap with custom filters (e.g. for CID blocking) Custom interface
- implementations such as Pinner or DAGService
- Dynamic configuration of libp2p ...
One downside of this is that we're exposing the entire dependency
graph, init hooks, initialization, etc. to users, so this comes with a
caveat that we reserve the right to make breaking changes to the graph
structure and initialization logic (although this historically happens
rarely). If these things are changed, we should mention them in
release notes and changelogs though, since they could impact users of
this plugin interface.
I'm not particularly fond of DI frameworks (and neither are some of
the folks work on/near go-ipfs), but it seems unlikely that somebody
will rewrite the dependency wiring and lifecycle hooks of go-ipfs, and
add dynamic extension points, so this seems like a palatable
compromise.
There are also problems that we should clean up in how model the
go-ipfs app in fx, such as:
- We make extensive use of nested fx.Options, which fx itself
discourages because it "limits the user's ability to customize their
application". It should be easy to flatten these out into a single
[]fx.Option slice.
- We pass around a list of opaque libp2p opts, which makes it hard to
customize after-the-fact...we should consider naming each of these
opts and providing them to fx as proper dependencies, so that they can
be explicitly overridden.
- We call fx.Invoke() in some places with anonymous functions. We
should instead only pass exported functions to fx.Invoke(), so that
they have exported names, which would make it easier to remove/augment
the invocations that happen when the app is initialized.
These aren't blocking issues, they just make it harder and more
brittle to customize go-ipfs with this plugin.
This preloads the peerlog plugin in the ipfs binary, but keeps it
disabled by default. To enabled it, set Enabled=true in its config.
The motivation is to simplify building and deploying gateways, and for
them to use binaries that are more similar to release bins.
* feat: switch to using go-ipld-prime for codecs, path resolution, and the `dag put/get` commands
* fix: `dag put/get` not roundtripping due to an extra new line being added (https://github.com/ipfs/go-ipfs/issues/3503)
More detailed information is in the CHANGELOG.md file. Very high level:
* IPLD codecs (and their plugins) must use go-ipld-prime
* Added support for the dag-json codec
* `dag get/put` use IPLD codec names from the multicodec table
* `dag get` defaults to dag-json output instead of json, but may output with other codecs
* Data model pathing can be achieved using the /ipld prefix. For example, you can use `/ipld/QmFoo/Links/0/Hash` to traverse through a DagPB node
* With `dag get/put` the DagPB field names have been changed to match the ones in the protobuf listed in the specification
Co-authored-by: hannahhoward <hannah@hannahhoward.net>
Co-authored-by: Daniel Martí <mvdan@mvdan.cc>
Co-authored-by: acruikshank <acruikshank@example.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Will Scott <will.scott@protocol.ai>
Co-authored-by: Will Scott <will@cypherpunk.email>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Eric Myhre <hash@exultant.us>
Avoid ever blocking new connections in the peer logger. Instead:
1. Send all new peers to a highly buffered channel.
2. Emit "dropped event" errors whenever we detect that we're dropping events and falling behind.
3. Don't log protocols, they're too large.
4. Don't log disconnects, we don't need them.
- allow IPFS_PLUGINS build-arg to be passed to docker build command
- remove annoying logging from peerlog plugin that appears multiple times in docker logs
usage:
docker build --build-arg IPFS_PLUGINS="foo bar baz"
License: MIT
Signed-off-by: Oli Evans <oli@tableflip.io>
adds a plugin to log just peerIDs for nodes we connect to, to allow us to see how many unique peers a node connects to over time.
It's set up a preloaded plugin, so it would be enabled by default. We may want to extract it to a seperate repo so we can make it an optional add on.
**Usage**
```console
$ GOLOG_FILE=~/peer.log IPFS_LOGGING_FMT=json ipfs daemon
```
**output**
```json
{"level":"info","ts":"2020-02-10T13:54:26.639Z","logger":"plugin/peerlog","caller":"peerlog/peerlog.go:51","msg":"connected","peer":"QmS2H72gdrekXJggGdE9SunXPntBqdkJdkXQJjuxcH8Cbt"}
{"level":"info","ts":"2020-02-10T13:54:59.095Z","logger":"plugin/peerlog","caller":"peerlog/peerlog.go:56","msg":"disconnected","peer":"QmS2H72gdrekXJggGdE9SunXPntBqdkJdkXQJjuxcH8Cbt"}
```
License: MIT
Signed-off-by: Oli Evans <oli@tableflip.io>
For now, configs specified in `daemon --init-config` and `init CONFIG` are not
available. We should fix this eventually but isn't necessary for now (and
supporting this will be annoying).
* Updates go-ipfs-cmds to try to get the tests to pass on travis.
* While we're at it, fix duplicate gx deps.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
