* fix/gateway: escape directory redirect URLs
When a director gets redirected to a URL with a treailing slash, special chars in the directory name must be escaped in the redirect URL. This upgrades to a version of box that has that fix.
Closes#10536
* Fix sharness test for new redirect URLs
* Update to latest boxo
* Use latest gateway-conformance
* feat(libp2p): enable shared TCP listeners
* docs: switch mentions of /ws to /tcp/4001
* feat: AutoTLS.AutoWSS
This adds AutoTLS.AutoWSS flag that is set to true by default.
It will check if Addresses.Swarm contain explicit /ws listener,
and if not found, it will append one per every /tcp listener
This way existing TCP ports are reused without any extra configuration,
but we don't break user's who have custom / explicit /ws listener
already.
I also moved logger around, to include Addresses.Swarm inspection
results in `autotls` logger.
* chore: go-libp2p v0.38.1
https://github.com/libp2p/go-libp2p/releases/tag/v0.38.0https://github.com/libp2p/go-libp2p/releases/tag/v0.38.1
* docs: AutoTLS.AutoWSS and go-libp2p v0.38.x
* chore: p2p-forge/client v0.2.0
https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.0
* fix: disable libp2p.ShareTCPListener() in PNET
* chore(ci): timeout sharness after 15m
average successful run is <9 minutes, no need to wait for 20
https://github.com/ipfs/kubo/actions/workflows/sharness.yml?query=is%3Asuccess
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: expose BlockKeyCacheSize and enable WriteThrough when bloom filter disabled
* import/config: add BatchMaxSize and BatchMaxNodes
* config: make BlockKeyCacheSize an OptionalInteger
* config: add and wire datastore.WriteThrough option
* config: omitempty on BlockKeyCacheSize
* changelog: rewrite entry about new options for the datastore
* config: add docs for BatchMaxNodes and BatchMaxSize
* config: make WriteThrough an optional Flag
* changelog: improve description of new datastore/import options
* refactor: DefaultWriteThrough as bool
* chore: boxo v0.26.0
* docs: config and changelog fixes
* cmd/files: flush parent folders
This is a mitigation to increased MFS memory usage in the course of many writes operations.
The underlying issue is the unbounded growth of the mfs directory cache in
boxo. In the latest boxo version, this cache can be cleared by calling Flush()
on the folder. In order to trigger that, we call Flush() on the parent folder
of the file/folder where the write-operations are happening.
To flushing the parent folder allows it to grow unbounded. Then, any read
operation to that folder or parents (i.e. stat), will trigger a sync-operation to match
the cache to the underlying unixfs structure (and obtain the correct node-cid).
This sync operation must visit every item in the cache. When the cache has grown too much,
and the underlying unixfs-folder has switched into a HAMT, the operation can take minutes.
Thus, we should clear the cache often and the Flush flag is a good indicator
that we can let it go. Users can always run with --flush=false and flush at
regular intervals during their MFS writes if they want to extract some performance.
Fixes#8694, #10588.
* cmd/files: docs and changelog for --flush changes
* pinmfs: mitigate slow mfs writes when it triggers
This mitigates slow mfs writes when the pinmfs daemon calls mfs.RootNode()
When writing lots of files to MFS, this call triggers a mfs directory cache
sync operations. The cache grows forever and is unbounded. The more files
added to a directory, the longer it takes. In the meantime, writing to mfs is
locked.
The pinmfs, even when no remote pinning services are configured, will trigger
this issue. When RootNode() takes more than 30 seconds, the issue will be triggered
continuously causing a write-deadlock onto MFS.
This commit does not fix the fact that if you write 1M items into an MFS
directory, the first time that the directory is traversed it will still have
to sync those 1M items. It does at least prevent writes from stalling after
about ~6000 items.
* pinmfs: fix test
The issue is that for dht providing to work correctly, at least two nodes must
have dht routing.
Before, dht providing in the test always failed, but the test succeeded when
the tested-CID was the one that the failure happened with (the first in the
list for ProvideMany).
* feat(bootstrap): /dnsaddr/va1.bootstrap.libp2p.io
this adds
https://github.com/libp2p/js-libp2p-amino-dht-bootstrapper
instance at /dnsaddr/va1.bootstrap.libp2p.io
to ensure Kubo does not depend on single language stack
for initial bootstrap
* docs: document where defaults live
* test: updated bootstrappers
* chore: update to boxo without goprocess
* Use boxo fix for registering metrics
* chore: switch to boxo main with PR 723
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
