Upgrade to the latest go-f3 and allow F3 chain exchange topics
Upgrade to the latest `go-f3` and add the F3 chain exchange topics to
the allowed topic list for both static and dynamic manifests.
Fixes https://github.com/filecoin-project/go-f3/issues/809
The initial implementation of IPNI used GossipSub as a way to propagate
IPNI advertisement chain. To do this the propagation had to be relayed
through the Lotus node due to strict Filecoin GossipSub validation
rules.
Since then IPNI has moved on to roll out its own sync protocol that
works over HTTP, and HTTP-over-libp2p. This move has been the official
way of advertising content to IPNI federation over a year now.
Therefore, remove the ad relay over pubsub via Lotus node as it is now
considered to have reached its EOL as a mechanism for advertising to
IPNI.
* chain index complete for msgs and txns
* dont need observer changes for now
* changes
* fix tests
* fix tests
* use th right context
* index empty tipsets correctly
* implement automated backfilling
* add event indexing and remove all old indices
* fix test
* revert deployment test changes
* revert test changes and better error handling for eth tx index lookups
* fix sql statments naming convention
* address review for Index GC
* more changes as per review
* changes as per review
* fix config
* mark events as reverted during reconciliation
* better reconciliation; pens down and code complete; also reconcile events
* fix tests
* improve config and docs
* improve docs and error handling
* improve read logic
* improve docs
* better logging and handle ennable event storage
* improve logs and index init proc
* better logging
* fix bugs based on calibnet testing
* create sqliite Indices
* gc should be based on epochs
* fix event query
* foreign keys should be enabled on the DB
* reverted tipsets should be removed as part of GC
* release read lock
* make it easy to backfill an empty index using reconciliation
* better docs for reconciliation
* fix conflicts with master
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix go mod
* fix formatting
* revert config changes
* address changes in observer
* remove top level chainindex package
* changes as per review
* changes as per review
* changes as per review
* handle index with reverted tipsets during reconciliation
* changes as per review
* fix type of max reconcile epoch
* changes to reconciliation as per review
* log ipld error
* better logging of progress
* disable chain indexer hydrate from snapshot based on config
* always populate index
* make config easy to reason about
* fix config
* fix messaging
* revert config changes
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes as per review
* make error messages homogenous
* fix indentation
* changes as per review
* feat: recompute tipset to generate missing events if event indexing is enabled (#12463)
* auto repair events
* make jen
* fix leaky abstraction
* better docs for gc retention epoch
* imrpove DB handling (#12485)
* fix conflict
* fix lite node config for indexer
* exclude reverted events from eth get logs if client queries by epoch
* Simply addressing for event lookups in the index.
simply addressing for event lookups
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix tests
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` (#12450)
* fix conflicts with chain indexer
* feat: chain indexer todos [skip changelog] (#12462)
* feat: finish todos of validation api
* feat: add indexed data verification with chain store
* feat: address comments and finish TODO
* fix: build issue
* address comments
* fix: ci issue
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes to Index Validation API based on Rodds first review
* build chain indexer API
* improve error handling
* feat: lotus-shed tooling for chain indexer (#12474)
* feat: add lotus-shed command for backfilling chain indexer
* feat: add lotus-shed command for inspecting the chain indexer
* feat: use single lotus-shed command to inspect and backfill
* fix: remove the unused queries
* small changes
* add change log
* backfilling improvements and fixes
* finish chain index validation and backfill tooling
* user documentation for the
* validate from epoch
* Apply suggestions from code review
Suggestions from Steve's read of the user doc.
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes to user doc as per review
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes to user doc as per review
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes as per review
* feat: add event entries count in validation API (#12506)
* feat: add event entry count in validation API
* address comments
* use sqllite defaults (#12504)
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* write chain index to a different dir
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* fix conflicts
* UX improvements to backfilling
* feat: tests for the chain indexer (#12521)
* ddl tests
* tests for the chain indexer
* finish unit tests for chain indexer
* fix formatting
* cleanup reverted tipsets to avoid db bloat
* fix logging
* test for filter by address
* test gc cascade delete
* fix db locked error during backfilling
* fix var name
* increase db locked timeout
* fix db locked issue
* reduce db lock timeout
* no lock in gc
* reconcile does not need lock
* improved error handling
* Update chain-indexing-overview-for-rpc-providers.md
Doc updates based on @jennijuju feedack.
* Update chain-indexing-overview-for-rpc-providers.MD
Fixes after reviewing 33c1ca1831
* better metrics for backfilling
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* tests for changes to event addressing
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes as per review -> round 1
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* log tipset key cid
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix docs
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix tests
* fix tests
* make jen
* fix conflicts
---------
Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* fix lint
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* remove reverted flag from RPC
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix testing of events and dummy chain store
* remove lotus shed commands for old Indices
* change type of event counts to uint64
* only recompute events if theyre not found
* short-circuit empty events path for older tipsets
* chain indexer must be enabled if ETH RPC is enabled
* change name of message_id column to id in tipset_message table
* only expose SetRecomputeTipSetStateFunc
* dont block on head indexing for reading messages
* document why we're only checking for missing events for a single tipset
* document when we query for reverted events
* simplify event collection
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix test
* change event_id to id in the event table
* change head indexed timeout
* remove deprecated config options
* fail ETH RPC calls if ChainIndexer is disabled
* fix docs
* remove the tipset key cid func from lotus shed
* address review comments
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* chore(events): remove unnecessary DisableRealTimeFilterAPI (#12610)
* feat(cli): add --quiet to chainindex validate-backfill + cleanups (#12611)
* fix tests
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* error type for disabled chainindexer
* fix(chainindex): recompute tipset when we find no receipts
* fix(chainindexer): backfilling should halt when chain state data is missing and not backfill parents (#12619)
* fix backfilling UX
* Update chain/index/api.go
Co-authored-by: Rod Vagg <rod@vagg.org>
* address review
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
* reduce log noise
* make jen
* make jen
* docs: finishing chain-indexer-overview-for-operators.md (#12600)
* Followup to PR #12450 for doc updates
This is being used to resolve the unresolved items in https://github.com/filecoin-project/lotus/pull/12450 since that PR is unwieldly at this point.
* Incorporated some items and added TODOs based on unresolved items from https://github.com/filecoin-project/lotus/pull/12450
* Incorporating more feedback
* Pointing to issue to learn about benefits
* Formatting fixes
* Apply most of the suggestions from @rvagg code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Incorporating feedback from https://github.com/filecoin-project/lotus/pull/12600#discussion_r1802519453
* Addressing https://github.com/filecoin-project/lotus/pull/12600#discussion_r1802540042 and more
* Moved chain-indexer docs to documentation
Renamed
Added ToC
We can move to lotus-docs later
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* Added upgrade path when importing chain state from a snapshot.
* Typo fixes
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* chore(doc): "regular checks" section for chainindexer docs (#12612)
* Apply suggestions from @rvagg code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Incorporating @aarshkshah1992 feedback
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* remove go mod replace
* remove unnecessary changes from CHANGELOG
* fix test
* compare events AMT root (#12632)
* fix(chainindex): retry transaction if database connection is lost (#12657)
* retry database lost connection
* log context cancellation
* address review
* fix gateway itest: no chainindexer for lite nodes
* fix changelog
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network>
Co-authored-by: Steve Loeppky <biglep@filoz.org>
1. Move it next to all the F3 code.
2. Make it look more like a "service" so I won't have to duplicate the
start/stop code in the itests.
3. Rename participator to participant. Not perfect, but it's a word.
4. Make all the constants public.
5. Only depend on the parts of the API we need (will help with unit testing).
1. Instead of hard-coding the renewal to be within 2 instances of the
end of the lease, specify the renewal to be within 1/2th of the end of
the lease, rounded up (towards the end of the lease).
2. Instead of hard-coding the lease check interval to be
10s (problematic for tests where the block time is 100ms), set it to be
the minimum amount of time we expect to have to wait for the remaining
instances to complete.
In _theory_ a lotus node can return no manifest from `F3Manifest`. I
don't think that happens in practice but we should handle the case and
start over at the top (try to get another lease). If we really have no
manifest, lotus will tell us that F3 isn't ready yet and we'll backoff
for a bit.
* fix(f3): fix hot loop in F3 participation
* fix(f3): Correct lease expiration log message
* feat(f3): log not-before and not-after for new leases
* test(f3): make the f3-enabled test wait for at one lease to expire
---------
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
* Implement ticket based F3 participation lease
Implemented enhanced ticket-based participation system for F3 consensus
in `F3Participate`. This update introduces a new design where
participation tickets grant a temporary lease, allowing storage
providers to sign as part of the F3 consensus mechanism. This design ensures that
tickets are checked for validity and issuer alignment, handling errors
robustly. If there's an issuer mismatch, the system advises miners to
retry with the existing ticket. If the ticket is invalid or expired,
miners are directed to obtain a new ticket via
`F3GetOrRenewParticipationTicket`.
Fixes https://github.com/filecoin-project/go-f3/issues/599
* Use fresh timer every time for F3 backoffs
To avoid potential of deadlock in case f3Participator is used from
multiple goroutines use throw-away timers at the price of higher GC.
Also use the cancel function in context explicitly in a unified stop
hook that awaits the participation to end before exiting.
* Strictly require start instance to never decrease
Require the start instance of a participation to never decrease if there
is an existing lease by the miner.
* feat(f3): update go-f3 to 0.7.0 and adapt for changes to the API
* feat(f3): Include the network name in the lease
That way we don't re-use leases across networks. It's a bit racy (we ask
for the manifest before we ask for the current progress) but it should
be fine because at least we won't create a lease for the new network
with a future instance.
There's still an ABA problem if we rapidly switch back and forth between
two networks but... let's just not do that? At least for the mainnet
switchover, that won't be an issue because we enforce a 900 epoch
silence period.
I have to say, I'm not happy about this. But... we can probably just
hard-code it in the future once we get rid of the dynamic manifest.
* Handle not ready error gracefully in participator
Back off and get a fresh token if F3 is not ready.
---------
Co-authored-by: Steven Allen <steven@stebalien.com>
This patch:
1. Bootstraps F3 when we hit the `F3BootstrapEpoch` (when non-negative).
2. Refuses any/all dynamic manifests once we get within one finality of said epoch.
3. Sets the F3 network name for mainnet to "filecoin".
4. Refuses any/all dynamic manifests that don't start with the expected
network name prefix.
* test(mpool): more complete testing of MpoolPush* methods
* feat(eth): add EthSendRawTransactionUntrusted, use via gateway eth_sendRawTransaction
* fix(eth): move EthSendRawTransactionUntrusted off main EthModuleAPI
- Move it to EthAPI so the GatewayAPI doesn't have to implement it.
This exposes bandwidth metrics via async callback to avoid
allocating/reporting metrics on any hot-paths. I'm using open telemetry
as we've already setup a bridge for F3 and opencensus is deprecated in
favor of open telemetry (so we're going to slowly move over anyways).
* use event index for receipt logs
* tests
* increment event count if there is an address mismatch
* itests for receipt logs consistency
* update ChangeLog
* changes as per review
* do not increment event count if address resolution fails
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* address review
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
This is a large diff, yet should have exactly zero functional changes
Ideally as a result of this some parts of the depchain will become lighter,
with downstream reaping the same benefits as the team that initiated this split.
P.S. work was done while forming better intuition of current dependency graph
storage/pipeline.NewPreCommitBatcher and storage/pipeline.New now have an additional
error return to deal with errors arising from fetching the sealing config.
* F3-370: integrate F3 dynamic manifest
* F3-370: make linter happy
* Set manifest sender identities
* Decode manifest sender peer ID from string before using it
Peer ID is of type string internally but the internal string
representation is not the same as the encoded string representation.
Therefore, the latter needs to be decoded and cannot be casted to the
former. Otherwise, it will represent a different ID.
* `make gen` the pain of my life
---------
Co-authored-by: Masih H. Derkani <m@derkani.org>
* F3-388: add env var to disable F3
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* F3-388: check env variable in pubsub also
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* F3-388: use DISABLE_F3=1 instead of _yes_ like in drand
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Use IsF3Enabled() and remove the runtime stopping
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
---------
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Co-authored-by: Jakub Sztandera <oss@kubuxu.com>
Integrate F3 into Lotus
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Co-authored-by: Jennifer Wang <jiayingw703@gmail.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>
Co-authored-by: Steven Allen <steven@stebalien.com>
* remove client CLI
* remove markets CLI from miner
* remove markets from all CLI
* remove client API
* update go mod
* remove EnableMarkets flag
* remove market subsystem
* remove dagstore
* remove index provider
* remove graphsync and data-transfer
* remove markets
* go mod tidy
* fix cbor gen deps
* remove deal making from config
* remove eol alert
* go mod tidy
* changes as per review
* make jen
* changes as per review
* merge master
* remove libp2p from config
* miner does not have libp2p conn in api test
Closes: https://github.com/filecoin-project/lotus/issues/11679
* Introduce a `moved:"To.New.Config"` tag which prints a stderr warning when
you use one of these, but will move any set value to the new location if the
new location isn't already set itself.
* Look for `X is DEPRECATED` to hold certain fields back from documentation.
* Use `toml:"omitempty"` to prevent the default config output from having these
deprecated values.
