cc031163d9
fix (zip): handle offline edge case
2025-08-04 23:26:38 +10:00
a54fa5da00
fix (s3): default region for s3
2025-06-12 00:54:22 +10:00
2616c078b5
feature (s3): add timeout option for very large buckets
2025-06-10 21:42:17 +10:00
fb29c7b6c2
feature (offline): handle offline files
2025-06-05 12:58:09 +10:00
d5628fd03d
fix (s3): region issue on s3
...
for server like versitygw which as of today doesn't support
getBucketLocation, we'd be sending the wrong region through
2025-05-12 21:05:17 +10:00
084bb33e83
fix (s3): handle cancellation
2025-04-09 14:42:21 +10:00
21c8a7f597
fix (s3): assume role on ecs
2025-02-21 17:26:00 +11:00
82dd3adf5d
fix (versitygw): better handling of missing filesize
...
* fix (s3): bug on versitygw
the plugin would break on versitgw which on some version isn't sending
the size metadata, ending with a null pointer dereference.
I wasn't able to replicate this on my own but the bug report shows:
2025/01/10 04:35:47 http: panic serving 130.194.66.248:54701: runtime error: invalid memory address or nil pointer dereference
goroutine 5590 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1947 +0xbe
panic({0x19c2740?, 0x535ece0?})
/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls.func1(0xc000986bd0, 0xe0?)
/home/server/plugin/plg_backend_s3/index.go:218 +0x18a
github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc0005b01a8, {0x46ae758, 0xc000f187e0}, 0xc0006649a0, 0xc0005d7270, {0x0, 0x0, 0x0})
/home/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:7654 +0x1ad
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls({0xc0005b0178, 0xc00074cd80, 0xc000f188a0, {0x46ae758, 0xc000f187e0}, 0x32}, {0xc0008810f0?, 0x99?})
/home/server/plugin/plg_backend_s3/index.go:202 +0x31a
github.com/mickael-kerjean/filestash/filestash-enterprise/plugins/utils.wrapper.Ls({{0x46b77c0, 0xc000f18ff0}, {0x5511fc0, 0x0, 0x0}}, {0xc0008810f0?, 0x165dc9a?})
/home/filestash-enterprise/plugins/utils/authorisation.go:64 +0x94
github.com/mickael-kerjean/filestash/server/ctrl.FileLs(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/ctrl/files.go:136 +0x7a2
github.com/mickael-kerjean/filestash/server/middleware.LoggedInOnly.func1(0xc000001d40?, {0x46a8bc8, 0xc000f18810}, 0x0?)
/home/server/middleware/session.go:23 +0x2e
github.com/mickael-kerjean/filestash/server/middleware.SessionStart.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/session.go:77 +0x2e7
github.com/mickael-kerjean/filestash/server/middleware.WithPublicAPI.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:115 +0x3a7
github.com/mickael-kerjean/filestash/server/middleware.SecureOrigin.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:99 +0x4a2
github.com/mickael-kerjean/filestash/server/middleware.SecureHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:79 +0x28c
github.com/mickael-kerjean/filestash/server/middleware.ApiHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:22 +0x29d
github.com/mickael-kerjean/filestash/server.Build.NewMiddlewareChain.func18({0x46a8cb8, 0xc000904000}, 0xc000227680)
/home/server/middleware/index.go:28 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc000227540?, {0x46a8cb8?, 0xc000904000?}, 0x52f069?)
/usr/local/go/src/net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000480, {0x46a8cb8, 0xc000904000}, 0xc000227400)
/home/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
net/http.serverHandler.ServeHTTP({0xc000c41d40?}, {0x46a8cb8?, 0xc000904000?}, 0x6?)
/usr/local/go/src/net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc00059d950, {0x46ae758, 0xc000832240})
/usr/local/go/src/net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 135
/usr/local/go/src/net/http/server.go:3360 +0x485
* fix (s3): versitygw issue
* fix (uint64): size
* fix (s3): better handling of missing size info
2025-01-10 16:23:29 +11:00
29db7d6934
fix ( #799 ): s3 - versitygw bug
...
* fix (s3): bug on versitygw
the plugin would break on versitgw which on some version isn't sending
the size metadata, ending with a null pointer dereference.
I wasn't able to replicate this on my own but the bug report shows:
2025/01/10 04:35:47 http: panic serving 130.194.66.248:54701: runtime error: invalid memory address or nil pointer dereference
goroutine 5590 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1947 +0xbe
panic({0x19c2740?, 0x535ece0?})
/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls.func1(0xc000986bd0, 0xe0?)
/home/server/plugin/plg_backend_s3/index.go:218 +0x18a
github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc0005b01a8, {0x46ae758, 0xc000f187e0}, 0xc0006649a0, 0xc0005d7270, {0x0, 0x0, 0x0})
/home/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:7654 +0x1ad
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls({0xc0005b0178, 0xc00074cd80, 0xc000f188a0, {0x46ae758, 0xc000f187e0}, 0x32}, {0xc0008810f0?, 0x99?})
/home/server/plugin/plg_backend_s3/index.go:202 +0x31a
github.com/mickael-kerjean/filestash/filestash-enterprise/plugins/utils.wrapper.Ls({{0x46b77c0, 0xc000f18ff0}, {0x5511fc0, 0x0, 0x0}}, {0xc0008810f0?, 0x165dc9a?})
/home/filestash-enterprise/plugins/utils/authorisation.go:64 +0x94
github.com/mickael-kerjean/filestash/server/ctrl.FileLs(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/ctrl/files.go:136 +0x7a2
github.com/mickael-kerjean/filestash/server/middleware.LoggedInOnly.func1(0xc000001d40?, {0x46a8bc8, 0xc000f18810}, 0x0?)
/home/server/middleware/session.go:23 +0x2e
github.com/mickael-kerjean/filestash/server/middleware.SessionStart.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/session.go:77 +0x2e7
github.com/mickael-kerjean/filestash/server/middleware.WithPublicAPI.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:115 +0x3a7
github.com/mickael-kerjean/filestash/server/middleware.SecureOrigin.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:99 +0x4a2
github.com/mickael-kerjean/filestash/server/middleware.SecureHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:79 +0x28c
github.com/mickael-kerjean/filestash/server/middleware.ApiHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:22 +0x29d
github.com/mickael-kerjean/filestash/server.Build.NewMiddlewareChain.func18({0x46a8cb8, 0xc000904000}, 0xc000227680)
/home/server/middleware/index.go:28 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc000227540?, {0x46a8cb8?, 0xc000904000?}, 0x52f069?)
/usr/local/go/src/net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000480, {0x46a8cb8, 0xc000904000}, 0xc000227400)
/home/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
net/http.serverHandler.ServeHTTP({0xc000c41d40?}, {0x46a8cb8?, 0xc000904000?}, 0x6?)
/usr/local/go/src/net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc00059d950, {0x46ae758, 0xc000832240})
/usr/local/go/src/net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 135
/usr/local/go/src/net/http/server.go:3360 +0x485
* fix (s3): versitygw issue
* fix (uint64): size
2025-01-10 16:13:33 +11:00
0d8ea12517
fix (s3): bug on versitygw ( #798 )
...
the plugin would break on versitgw which on some version isn't sending
the size metadata, ending with a null pointer dereference.
I wasn't able to replicate this on my own but the bug report shows:
2025/01/10 04:35:47 http: panic serving 130.194.66.248:54701: runtime error: invalid memory address or nil pointer dereference
goroutine 5590 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1947 +0xbe
panic({0x19c2740?, 0x535ece0?})
/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls.func1(0xc000986bd0, 0xe0?)
/home/server/plugin/plg_backend_s3/index.go:218 +0x18a
github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc0005b01a8, {0x46ae758, 0xc000f187e0}, 0xc0006649a0, 0xc0005d7270, {0x0, 0x0, 0x0})
/home/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:7654 +0x1ad
github.com/mickael-kerjean/filestash/server/plugin/plg_backend_s3.S3Backend.Ls({0xc0005b0178, 0xc00074cd80, 0xc000f188a0, {0x46ae758, 0xc000f187e0}, 0x32}, {0xc0008810f0?, 0x99?})
/home/server/plugin/plg_backend_s3/index.go:202 +0x31a
github.com/mickael-kerjean/filestash/filestash-enterprise/plugins/utils.wrapper.Ls({{0x46b77c0, 0xc000f18ff0}, {0x5511fc0, 0x0, 0x0}}, {0xc0008810f0?, 0x165dc9a?})
/home/filestash-enterprise/plugins/utils/authorisation.go:64 +0x94
github.com/mickael-kerjean/filestash/server/ctrl.FileLs(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/ctrl/files.go:136 +0x7a2
github.com/mickael-kerjean/filestash/server/middleware.LoggedInOnly.func1(0xc000001d40?, {0x46a8bc8, 0xc000f18810}, 0x0?)
/home/server/middleware/session.go:23 +0x2e
github.com/mickael-kerjean/filestash/server/middleware.SessionStart.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/session.go:77 +0x2e7
github.com/mickael-kerjean/filestash/server/middleware.WithPublicAPI.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:115 +0x3a7
github.com/mickael-kerjean/filestash/server/middleware.SecureOrigin.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:99 +0x4a2
github.com/mickael-kerjean/filestash/server/middleware.SecureHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:79 +0x28c
github.com/mickael-kerjean/filestash/server/middleware.ApiHeaders.func1(0xc000001d40, {0x46a8bc8, 0xc000f18810}, 0xc000227680)
/home/server/middleware/http.go:22 +0x29d
github.com/mickael-kerjean/filestash/server.Build.NewMiddlewareChain.func18({0x46a8cb8, 0xc000904000}, 0xc000227680)
/home/server/middleware/index.go:28 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc000227540?, {0x46a8cb8?, 0xc000904000?}, 0x52f069?)
/usr/local/go/src/net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000480, {0x46a8cb8, 0xc000904000}, 0xc000227400)
/home/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
net/http.serverHandler.ServeHTTP({0xc000c41d40?}, {0x46a8cb8?, 0xc000904000?}, 0x6?)
/usr/local/go/src/net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc00059d950, {0x46ae758, 0xc000832240})
/usr/local/go/src/net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 135
/usr/local/go/src/net/http/server.go:3360 +0x485
2025-01-10 15:52:28 +11:00
be716e6cf9
fix (time): hide time whenever not available
2024-04-22 01:05:20 +10:00
24e7b5eb35
fix (plg_search_sqlitefts): prevent context expiration if possible
2024-01-31 01:01:35 +11:00
4660f3a2ff
feature (plg_backend_s3): calculate the right mime type
2024-01-25 21:44:55 +11:00
def45d5ad5
feature (s3): multithread on s3 plugin
2023-11-27 00:25:28 +11:00
8c4ca9da76
fix (plg_backend_s3): revert #584 after broken e2e
...
This reverts commit 67e2a3485307f7a7bbf0761da87d2ad9fb9a2e3d.
2023-04-10 10:56:40 +10:00
67e2a34853
feature (plg_backend_s3): add param for forcePathStyle - #584
...
Co-authored-by: xiayan.cn <xiayan.cn@bytedance.com>
Co-authored-by: Mickael <mickael@kerjean.me>
2023-04-09 13:10:33 +10:00
586e803bde
feature (plg_backend_s3): access S3 via AWS role - #583
...
Co-authored-by: Daniel Gomes-Sebastiao <daniel.gomes@adscale.co.nz>
2023-04-09 13:07:14 +10:00
091e9c0c78
fix(s3) save #577
...
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
2023-04-01 10:53:29 +11:00
fa9f1bc3e4
fix (ls): configurable ls timeout for plugins
2023-03-24 22:37:19 +11:00
a57a4a7e15
fix (s3): exit s3 list via IsTruncated flag
...
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
2023-03-24 10:53:38 +11:00
62b066feab
fix ( #559 ): handle large amount of deletion on s3
...
Signed-off-by: zhucan <zhucan.k8s@gmail.com>
2023-02-21 17:31:10 +11:00
922d8fecc3
fix ( #545 ): emergency fix - credit and original author @microyahoo
2023-02-08 15:23:49 +11:00
3727981e75
fix (s3): cloudflare s3 default
2023-01-26 01:20:39 +11:00
fb4a6a7ab2
fix (s3): stop condition for recursive ls in s3
2022-12-08 16:05:40 +11:00
e7bb5b052a
feature (s3): s3 secret key as password field
2022-11-25 16:39:10 +11:00
dec9e2f60d
fix (s3): s3 rename - #421
...
* fix MV function in the S3 backend
* fix space encoding
Co-authored-by: Quentin Bramas <bramas@unistra.fr>
2022-02-06 20:29:35 +11:00
bcf5c467c1
feature (viewer): support wave file
2021-11-24 23:31:30 +11:00
1f447dc7d0
fix (s3): moving files - no such key error
2021-11-24 23:24:30 +11:00
edbf16871c
fix ( #410 ): error when moving files with the S3 backend
...
* url encode paths when moving object in the S3 backend
* fix renaming with the same name
Co-authored-by: Quentin Bramas <bramas@unistra.fr>
2021-09-27 20:06:09 +10:00
110280e94b
improve (s3): faster whenever using role based binding on ec2
2021-09-27 19:59:11 +10:00
1c337f5c57
improve (log): additional debugging
2021-09-14 22:37:37 +10:00
3214cfd588
feature (s3): add ec2 role base authentication
2021-09-12 17:16:32 +10:00
0360412444
feature (s3): implement mv - #405
...
* WIP: implement s3 recursive Mv
* fix recursive move, add logs
2021-08-30 23:48:20 +10:00
136afbeac2
improve (s3): support for prefixes with > 1k objects ( #395 )
...
Previously filestash would only show the first 1000 objects in a given
s3 prefix.
We solve this by paginating the ListObjectsV2 responses and iterating
over all pages instead of just the first one.
Tested by manually connecting to a private s3 bucket with multiple
thousands of objects under a single prefix and verifying that all
objects are visible in the UI.
2021-05-25 22:00:23 +10:00
d6ad96f9f4
fix (S3): use ListObjectsV2 - #334
2020-10-24 00:49:54 +11:00
6adf96d66e
feat (s3): adds session token to config ( #306 )
2020-09-15 19:36:35 +10:00
6667725e12
fix (s3): moving folder isn't unsupported
2020-06-23 22:43:53 +10:00
e0b8bf2e70
fix (s3): rm merge
2020-06-23 21:45:42 +10:00
ade3c332e8
fix (s3): s3 rm bug - #228
2020-06-23 21:25:16 +10:00
a40c2f7a53
fix (s3): error when path in ls doesn't exist
2020-06-23 18:56:40 +10:00
8f1097273e
maintain (s3): migrate s3 as a plugin
2020-06-23 18:15:12 +10:00
