When all blocks in a trace_filter range are null rounds, the API was returning
JSON `null` instead of `[]` because the results slice was never initialised.
Alias for F3GetLatestCertificate -> ECChain.TipSets[-1] -> TipSet but with
fall-back to EC when F3 is not properly functioning or is behind EC.
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* feat(frc-0108): intro SnapshotMetadata
* feat(frc-0108): import and export for snapshotV2
* test(frc-0108): add TestChainExportImportWithF3Data
* chore: make lint happy
* chore: fix log args & clean unused FIXME
* fix: generate SnapshotMetadata via WriteMapEncodersToFile
* chore: bump go-f3 from v0.8.9 to v0.8.10
* fix: rename SnapshotMetadata.HeadTipsetKeys => SnapshotMetadata.HeadTipsetKey
* chore: batter log for snapshot
* chore: update go.sum
* opt: load f3 snapshot into temp file first
* chore: fix make lint
* docs: add CHANGELOG entry for F3-aware snapshot v2 support
Add comprehensive CHANGELOG entry documenting the new snapshot v2 format
that includes embedded F3 finality certificates per FRC-0108. This
significantly improves node synchronization time by reducing F3 catchup
from ~8 hours.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: roots directly contains the tipset CIDs for v1 format
* opt: get NetworkName via buildconstants.F3Manifest()
---------
Co-authored-by: tedious <tedious@gmail.com>
Co-authored-by: Steve Loeppky <biglep@filoz.org>
Co-authored-by: Claude <noreply@anthropic.com>
* fix#13247
* update CHANGELOG.md
* don't need to escape _ inside ticks
* fix eth_getStorageAt with CallOnState
* update changelog: eth_getStorageAt
* also use CallOnState for EthGetCode; can now revert From address change
* dep(f3): update go-f3
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix(f3/cli/api): add F3GetPowerTableByInstance
This allows resolution of historical power tables.
The other methods require chain state to retrieve the power table.
The certstore stories all power tables diffs and is able to provide a
power table at artbitrary instance.
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* mod tidy
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* add changelog
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix f3 tests
the fix is to set bootstrapEpoch = Finality
this is because of short block times and large delay in production of
blocks following the block 0
this causes f3 to try to start up too early
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Revert "fix f3 tests"
This reverts commit 0f8033a80f.
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Try out the bootstrap fix
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* make docsgen-cli
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Expose F3GetPowerTableByInstance on gateway
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Update go-f3 to 0.8.8
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* Fix manifest template to not use non-existent fields
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
---------
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* feat: network version 27 skeleton
* fix(nv27): skip TockFix actors version sanity check
As per comment inline, this is getting in the way of landing a skeleton
for nv27 with an identical actors bundle so version resolution for v16 CIDs
was showing 17.
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
* feat(paych): add EnablePaymentChannelManager option & disable paych by default
This change introduces a new configuration option EnablePaymentChannelManager to control whether the payment channel manager is started on node startup.
- Added EnablePaymentChannelManager boolean field to FullNode config struct
- Set default value to false (disabled by default) since payment channels get minimal use on mainnet
- Added conditional wiring in node/builder_chain.go to only start payment channel manager when config option is enabled
- Updated config documentation with clear description of the option
This saves resources for light nodes and most full nodes that don't use payment channel functionality, while still allowing users to enable it when needed by setting EnablePaymentChannelManager: true in their config.
docs: add documentation for EnablePaymentChannelManager config option
- Updated default-lotus-config.toml with proper documentation for the new EnablePaymentChannelManager option
docs: update CHANGELOG.md with PR number #13139
Updated the CHANGELOG.md entry for EnablePaymentChannelManager feature with the actual PR number from the created pull request.
fix: address reviewer feedback on CHANGELOG.md - Fix PR number reference from PR_NUMBER to 13139 - Remove unintended formatting changes as requested by reviewer
feat(paych): fix gen-check
docs: update config documentation after reordering EnablePaymentChannelManager field
* feat(paych): make disabled paych work & tests pass
fix(paych): add DisabledPaychAPI stub to return errors when external paych API is called
fix(cfg): introduce PaymentChannels top-level config option
fix(test): make integration tests work with paych disabled
* feat(paych): rename duplicate PaychAPI to avoid confusion
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
Assert that EC finalized tipset height is never negative in v2 APIs
Fix an edge case where if the head epoch is below the chain finality
policy (defaulting to 900) EC finalized head height would become
negative.
Fixes: #13167
Fixes a problem introduced, but overlooked in
https://github.com/filecoin-project/lotus/pull/13123
which meant that F3 was never wired into the v2 TipSetProvider. Integration
tests failed but we ignored it because we had a common flaky that was related
so didn't take notice of it.
Disable F3 module when lotus is run with `--lite` commend since in that
mode the daemon is simply a reverse proxy of a full node.
Stop using F3Backend directly in Eth modules (#13127)
* Fall back to EC finalized tipset if F3 is too far behind in eth APIs
Update the Eth v2 APIs to fall back to EC finalized tipset if F3 is too
far behind.
* Update changelog
* Fall back to EC if F3 finalized tipeset is older than 900 epochs
In Lotus v2 APIs, in an event where F3 finalized tipset is too far
behind EC, > 900 epochs return EC finalized tipset.
Part of: https://github.com/filecoin-project/lotus/issues/13062
* Update changelog
* Implement `safe` tag in tipset selection
Implement the ability to select a tipset by tag `safe`. The safe tag
would select the earliest between finalised and heaviest - 200.
Implement the Lotus V2 APIs for :
* `StateGetActor` - enhanced to accept tipset selector.
* `StateGetID` - enhanced to accept tipset selector / renamed from
`LookupID` in v1 for consistency.
Tests are added to exercise the end-to-end API call through JSON rpc
with the raw wire format of the JSON-RPC request. Additional factory
functions are added for `TipSetLimit`, a new concept to specify a limit
for tipsets either as absolute height or relative to some
tipset selector. The future State message search APIs in v2 aim to use
Limit in upcoming PRs.
To unblock progress in making incremental progress at v2 apis I am
removing the implementation of StateSimulate and StateCompute in light
of discussions at #13028.
Introduce the first API for Lotus v2, focusing on `ChainGetTipSet`
within the `Chain` group. Define `TipSetSelector` for advanced
tipset retrieval options, and create a compact JSON-RPC call format.
Gracefully accommodate both EC and F3 finalized tipsets based on node
configuration, where:
* EC finalized tipset is returned when F3 is turned off, has no
finalized tipset or F3 isn't ready.
* F3 finalized is returned otherwise.
Support three categories of selectors under `TipSetSelector`:
* By tag: either "latest" or "finalized."
* By height: epoch, plus optional fallback to previous non-null tipset.
* By tipset key.
The selection falls back to tag "latest" if the user specifies no
selection criterion.
The JSON-RPC format is designed to use JSON Object as the parameters
passed to the RPC call to remain compact, and extensible.
Upgrade to go-f3 `v0.8.3`
Upgrade to the latest go-f3, to bubble up:
* various bug fixes
* extended manifest parameters for better control in passive testing
Note that the return type from F3GetProgress API has changed to now
include the input chain to an instance.
Update CLI templates to print the additional fields.
* refactor(api)!: remove SupportedProofTypes from StateGetNetworkParams
Breaking change: The SupportedProofTypes field has been removed from StateGetNetworkParams. This field was unreliable as it didn't match the FVM's actual supported proofs and was frequently overlooked during proof type updates
* feat: expose StateGetNetworkParams in Lotus Gateway API
feat: expose StateGetNetworkParams in Lotus Gateway API
* chore: update changelog
chore: update changelog
* chore: update codename for nv25 to "Teep"
* chore: set upgrade epoch for Calibration net
* chore: fix GenesisNetworkVersion for butterfly
* chore: set upgrade height for butterflynet
* chore: update to builtin-actors v16.0.0-rc1
* chore: update go-state-types to v0.16.0-rc1
* chore: update filecoin-ffi to v1.32.0-rc1
* chain index complete for msgs and txns
* dont need observer changes for now
* changes
* fix tests
* fix tests
* use th right context
* index empty tipsets correctly
* implement automated backfilling
* add event indexing and remove all old indices
* fix test
* revert deployment test changes
* revert test changes and better error handling for eth tx index lookups
* fix sql statments naming convention
* address review for Index GC
* more changes as per review
* changes as per review
* fix config
* mark events as reverted during reconciliation
* better reconciliation; pens down and code complete; also reconcile events
* fix tests
* improve config and docs
* improve docs and error handling
* improve read logic
* improve docs
* better logging and handle ennable event storage
* improve logs and index init proc
* better logging
* fix bugs based on calibnet testing
* create sqliite Indices
* gc should be based on epochs
* fix event query
* foreign keys should be enabled on the DB
* reverted tipsets should be removed as part of GC
* release read lock
* make it easy to backfill an empty index using reconciliation
* better docs for reconciliation
* fix conflicts with master
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix go mod
* fix formatting
* revert config changes
* address changes in observer
* remove top level chainindex package
* changes as per review
* changes as per review
* changes as per review
* handle index with reverted tipsets during reconciliation
* changes as per review
* fix type of max reconcile epoch
* changes to reconciliation as per review
* log ipld error
* better logging of progress
* disable chain indexer hydrate from snapshot based on config
* always populate index
* make config easy to reason about
* fix config
* fix messaging
* revert config changes
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes as per review
* make error messages homogenous
* fix indentation
* changes as per review
* feat: recompute tipset to generate missing events if event indexing is enabled (#12463)
* auto repair events
* make jen
* fix leaky abstraction
* better docs for gc retention epoch
* imrpove DB handling (#12485)
* fix conflict
* fix lite node config for indexer
* exclude reverted events from eth get logs if client queries by epoch
* Simply addressing for event lookups in the index.
simply addressing for event lookups
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix tests
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` (#12450)
* fix conflicts with chain indexer
* feat: chain indexer todos [skip changelog] (#12462)
* feat: finish todos of validation api
* feat: add indexed data verification with chain store
* feat: address comments and finish TODO
* fix: build issue
* address comments
* fix: ci issue
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes to Index Validation API based on Rodds first review
* build chain indexer API
* improve error handling
* feat: lotus-shed tooling for chain indexer (#12474)
* feat: add lotus-shed command for backfilling chain indexer
* feat: add lotus-shed command for inspecting the chain indexer
* feat: use single lotus-shed command to inspect and backfill
* fix: remove the unused queries
* small changes
* add change log
* backfilling improvements and fixes
* finish chain index validation and backfill tooling
* user documentation for the
* validate from epoch
* Apply suggestions from code review
Suggestions from Steve's read of the user doc.
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes to user doc as per review
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes to user doc as per review
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* changes as per review
* feat: add event entries count in validation API (#12506)
* feat: add event entry count in validation API
* address comments
* use sqllite defaults (#12504)
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* write chain index to a different dir
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* fix conflicts
* UX improvements to backfilling
* feat: tests for the chain indexer (#12521)
* ddl tests
* tests for the chain indexer
* finish unit tests for chain indexer
* fix formatting
* cleanup reverted tipsets to avoid db bloat
* fix logging
* test for filter by address
* test gc cascade delete
* fix db locked error during backfilling
* fix var name
* increase db locked timeout
* fix db locked issue
* reduce db lock timeout
* no lock in gc
* reconcile does not need lock
* improved error handling
* Update chain-indexing-overview-for-rpc-providers.md
Doc updates based on @jennijuju feedack.
* Update chain-indexing-overview-for-rpc-providers.MD
Fixes after reviewing 33c1ca1831
* better metrics for backfilling
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update chain/index/chain-indexing-overview-for-rpc-providers.MD
Co-authored-by: Rod Vagg <rod@vagg.org>
* tests for changes to event addressing
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* changes as per review -> round 1
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* log tipset key cid
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix docs
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix tests
* fix tests
* make jen
* fix conflicts
---------
Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steve Loeppky <biglep@filoz.org>
* fix lint
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* remove reverted flag from RPC
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix testing of events and dummy chain store
* remove lotus shed commands for old Indices
* change type of event counts to uint64
* only recompute events if theyre not found
* short-circuit empty events path for older tipsets
* chain indexer must be enabled if ETH RPC is enabled
* change name of message_id column to id in tipset_message table
* only expose SetRecomputeTipSetStateFunc
* dont block on head indexing for reading messages
* document why we're only checking for missing events for a single tipset
* document when we query for reverted events
* simplify event collection
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* fix test
* change event_id to id in the event table
* change head indexed timeout
* remove deprecated config options
* fail ETH RPC calls if ChainIndexer is disabled
* fix docs
* remove the tipset key cid func from lotus shed
* address review comments
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* chore(events): remove unnecessary DisableRealTimeFilterAPI (#12610)
* feat(cli): add --quiet to chainindex validate-backfill + cleanups (#12611)
* fix tests
* Apply suggestions from code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* error type for disabled chainindexer
* fix(chainindex): recompute tipset when we find no receipts
* fix(chainindexer): backfilling should halt when chain state data is missing and not backfill parents (#12619)
* fix backfilling UX
* Update chain/index/api.go
Co-authored-by: Rod Vagg <rod@vagg.org>
* address review
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
* reduce log noise
* make jen
* make jen
* docs: finishing chain-indexer-overview-for-operators.md (#12600)
* Followup to PR #12450 for doc updates
This is being used to resolve the unresolved items in https://github.com/filecoin-project/lotus/pull/12450 since that PR is unwieldly at this point.
* Incorporated some items and added TODOs based on unresolved items from https://github.com/filecoin-project/lotus/pull/12450
* Incorporating more feedback
* Pointing to issue to learn about benefits
* Formatting fixes
* Apply most of the suggestions from @rvagg code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Incorporating feedback from https://github.com/filecoin-project/lotus/pull/12600#discussion_r1802519453
* Addressing https://github.com/filecoin-project/lotus/pull/12600#discussion_r1802540042 and more
* Moved chain-indexer docs to documentation
Renamed
Added ToC
We can move to lotus-docs later
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* Added upgrade path when importing chain state from a snapshot.
* Typo fixes
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
* chore(doc): "regular checks" section for chainindexer docs (#12612)
* Apply suggestions from @rvagg code review
Co-authored-by: Rod Vagg <rod@vagg.org>
* Incorporating @aarshkshah1992 feedback
* Update documentation/en/chain-indexer-overview-for-operators.md
Co-authored-by: Rod Vagg <rod@vagg.org>
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* remove go mod replace
* remove unnecessary changes from CHANGELOG
* fix test
* compare events AMT root (#12632)
* fix(chainindex): retry transaction if database connection is lost (#12657)
* retry database lost connection
* log context cancellation
* address review
* fix gateway itest: no chainindexer for lite nodes
* fix changelog
---------
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network>
Co-authored-by: Steve Loeppky <biglep@filoz.org>
