* cmd/files: flush parent folders
This is a mitigation to increased MFS memory usage in the course of many writes operations.
The underlying issue is the unbounded growth of the mfs directory cache in
boxo. In the latest boxo version, this cache can be cleared by calling Flush()
on the folder. In order to trigger that, we call Flush() on the parent folder
of the file/folder where the write-operations are happening.
To flushing the parent folder allows it to grow unbounded. Then, any read
operation to that folder or parents (i.e. stat), will trigger a sync-operation to match
the cache to the underlying unixfs structure (and obtain the correct node-cid).
This sync operation must visit every item in the cache. When the cache has grown too much,
and the underlying unixfs-folder has switched into a HAMT, the operation can take minutes.
Thus, we should clear the cache often and the Flush flag is a good indicator
that we can let it go. Users can always run with --flush=false and flush at
regular intervals during their MFS writes if they want to extract some performance.
Fixes#8694, #10588.
* cmd/files: docs and changelog for --flush changes
* feat(cmds/add): --to-files option as files cp
* tests(to-files): ensure error handling is covered
this adds bunch of tests that guard UX around importing multiple files
into MFS, and the logic around trailing slash to indicate if the MFS
destination if a directory.
If the destination has a trailing slash, we ensure that the directory
exists and is a dir and not a file. this allows us to support
adding multipl files into MFS dir:
ipfs add file1.txt file2.txt --to-files /some/mfs/dir/
* docs: to-files helptext
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Unfortunately, API docs are genereted from go-ipfs sources
via https://github.com/ipfs/http-api-docs
so it all gets lost when API docs for new version are re-generated.
This re-applies manual fixes from:
https://docs.ipfs.io/reference/http/api/
so the next time http-api-docs are re-generated, those changes stay.
* Adds the `--parents` optiont to the first command so that it is runnable
* Updates the description to specifies the option to use if you _do_ want nonexistant intermediate directories created
- Make `ipfs files rm --force /nonexistant` succeed when the path does not exist.
- Add shaness test for removing nonexistant paths
- Refactor duplicated code to find a parent dir into a function
I've been writing scripts against the files api, and having to stat things before removing them is a pain. So this PR aims to make --force do what I'd expect it to.
License: MIT
Signed-off-by: Oli Evans <oli@tableflip.io>
Most of these are probably harmless but a few looked like they might actually be
bugs. Most of them are just faulty tests.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
