provider: buffer pin providers.

Fixes #10596.

The reproviding process can take long. Currently, each CID to be provided is
obtained by making a query to the pinner and reading one by one as the CIDs
get provided.

While this query is ongoing, the pinner holds a Read mutex on the pinset.

If a pin-add-request arrives, a goroutine will start waiting for a Write mutex
on the pinset. From that point, no new Read mutexes can be taken until the writer
can proceed and finishes.

However, no one can proceed because the read mutex is still held while the
reproviding is ongoing.

The fix is mostly in Boxo, where we add a "buffered" provider which reads the
cids onto memory so that they can be provided at its own pace without making
everyone wait.

The consequence is we will need more RAM memory. Rule of thumb is 1GiB extra per 20M cids to be reprovided.

(cherry picked from commit ba22102a640b3f41804319981d83908c96a96275)

core/node/provider.go

@@ -167,16 +167,23 @@ func newProvidingStrategy(onlyPinned, onlyRoots bool) interface{} {
 		IPLDFetcher fetcher.Factory `name:"ipldFetcher"`
 	}
 	return func(in input) provider.KeyChanFunc {
+		// Pinner-related CIDs will be buffered in memory to avoid
+		// deadlocking the pinner when the providing process is slow.
+
 		if onlyRoots {
-			return provider.NewPinnedProvider(true, in.Pinner, in.IPLDFetcher)
+			return provider.NewBufferedProvider(
+				provider.NewPinnedProvider(true, in.Pinner, in.IPLDFetcher),
+			)
 		}
 
 		if onlyPinned {
-			return provider.NewPinnedProvider(false, in.Pinner, in.IPLDFetcher)
+			return provider.NewBufferedProvider(
+				provider.NewPinnedProvider(false, in.Pinner, in.IPLDFetcher),
+			)
 		}
 
 		return provider.NewPrioritizedProvider(
-			provider.NewPinnedProvider(true, in.Pinner, in.IPLDFetcher),
+			provider.NewBufferedProvider(provider.NewPinnedProvider(true, in.Pinner, in.IPLDFetcher)),
 			provider.NewBlockstoreProvider(in.Blockstore),
 		)
 	}

docs/changelogs/v0.34.md

@@ -14,6 +14,7 @@
   - [Badger datastore update](#badger-datastore-update)
   - [Datastore Implementation Updates](#datastore-implementation-updates)
   - [One Multi-error Package](#one-multi-error-package)
+  - [Fix hanging pinset operations during reprovides](#fix-hanging-pinset-operations-during-reprovides)
   - [📦️ Important dependency updates](#-important-dependency-updates)
 - [👨‍👩‍👧‍👦 Contributors](#-contributors)
 
@@ -74,6 +75,10 @@ The go-ds-xxx datastore implementations have been updated to support the updated
 
 Kubo previously depended on multiple multi-error packages, `github.com/hashicorp/go-multierror` and `go.uber.org/multierr`. These have nearly identical functionality so there was no need to use both. Therefore, `go.uber.org/multierr` was selected as the package to depend on. Any future code needing multi-error functionality should use `go.uber.org/multierr` to avoid introducing unneeded dependencies.
 
+#### Fix hanging pinset operations during reprovides
+
+The reprovide process can be quite slow. In default settings, the reprovide process will start reading CIDs that belong to the pinset. During this operation, starvation can occurr for other operations that need pinset access (see https://github.com/ipfs/kubo/issue/10596). We have now switch to buffering pinset-related cids that are going to be reprovided in memory, so that we can free pinset mutexes as soon as possible so that pinset-writes and subsequent read operations can proceed. The downside is larger pinsets will need some extra memory, with an estimation of ~1GiB of RAM memory-use per 20 million items to be reprovided.
+
 #### 📦️ Important dependency updates
 
 - update `go-libp2p` to [v0.41.0](https://github.com/libp2p/go-libp2p/releases/tag/v0.41.0) (incl. [v0.40.0](https://github.com/libp2p/go-libp2p/releases/tag/v0.40.0))