To test it, set up an S3 bucket (in an AWS region that is not US
Standard, for read-after-write consistency), run `ipfs init`, then
edit `~/.ipfs/config` to say
"Datastore": {
"Type": "s3",
"Region": "us-west-1",
"Bucket": "mahbukkit",
"ACL": "private"
},
with the right values. Set `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` in the environment and you should be able to
run `ipfs add` and `ipfs cat` and see the bucket be populated.
No automated tests exist, unfortunately. S3 is thorny to simulate.
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
Earlier, it also checked checked the leveldb directory. That part
added no crash safety to the application, and just hardcoded
assumptions about the datastore.
If anything, this should rely on the absolute last item created by
fsrepo.Init, and there should be fsync guarantees about ordering.
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
dont GC blocks used by pinner
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
comment GC algo
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
add lock to blockstore to prevent GC from eating wanted blocks
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
improve FetchGraph
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
separate interfaces for blockstore and GCBlockstore
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
reintroduce indirect pinning, add enumerateChildren dag method
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
This commit improves (fixes) the FetchGraph call for recursively
fetching every descendant node of a given merkledag node. This operation
should be the simplest way of ensuring that you have replicated a dag
locally.
This commit also implements a method in the merkledag package called
EnumerateChildren, this method is used to get a set of the keys of every
descendant node of the given node. All keys found are noted in the
passed in KeySet, which may in the future be implemented on disk to
avoid excessive memory consumption.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
The addition of a locking interface to the blockstore allows us to
perform atomic operations on the underlying datastore without having to
worry about different operations happening in the background, such as
garbage collection.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
* ID service stream
* make the relay service use msmux
* fix nc tests
Note from jbenet: Maybe we should remove the old protocol/muxer
and see what breaks. It shouldn't be used by anything now.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
There was doublewrapping with an unneeded msgio. given that we
use a stream muxer now, msgio is only needed by secureConn -- to
signal the boundaries of an encrypted / mac-ed ciphertext.
Side note: i think including the varint length in the clear is
actually a bad idea that can be exploited by an attacker. it should
be encrypted, too. (TODO)
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
WARNING: No migration performed! That needs to come in a separate
commit, perhaps amended into this one.
This is the minimal rewrite, only changing the storage from
JSON(+extra keys) in Datastore to IPFS objects. All of the pinning
state is still loaded in memory, and written from scratch on Flush. To
do more would require API changes, e.g. adding error returns.
Set/Multiset is not cleanly separated into a library, yet, as it's API
is expected to change radically.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
sharness: Don't assume we know all things that can create garbage
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
Pinner had method GetManual that returned a ManualPinner, so every
Pinner had to implement ManualPinner anyway.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
These secondary copies were never actually queried, and didn't contain
the indirect refcounts so they couldn't become the authoritative
source anyway as is. New goal is to move pinning into IPFS objects.
A migration will be needed to remove the old data from the datastore.
This can happen at any time after this commit.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
OS X sed is documented as "-i SUFFIX", GNU sed as "-iSUFFIX". The one
consistent case seems to be "-iSUFFIX", where suffix cannot empty (or
OS X will parse the next argument as the suffix).
This used to leave around files named `refsout=` on Linux, and was
just confusing.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
This used to lead to large refcount numbers, causing Flush to create a
lot of IPFS objects, and merkledag to consume tens of gigabytes of
RAM.
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
