mirror of
https://github.com/rclone/rclone.git
synced 2025-12-11 22:14:05 +01:00
chunker: partially implement no-rename transactions (#4675)
Some storage providers e.g. S3 don't have an efficient rename operation. Before this change, when chunker finished an upload, the server-side copy and delete operations that renamed temporary chunks to their final names could take a significant amount of time. This PR records transaction identifier (versioning) in the metadata of chunker composite objects striving to remove the need for rename operations on such backends. This approach will be triggered be the new "transactions" configuration option, which can be "rename" (the default) or "norename". We implement the new approach for uploads (Put operations). The chunker Move operation still uses the rename operation of underlying backend. Filling this gap is left for a later PR. Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
This commit is contained in:
committed by
Ivan Andreev
parent
b029fb591f
commit
9cc8ff4dd4
@@ -151,6 +151,9 @@ Note that `list` assembles composite directory entries only when chunk names
|
||||
match the configured format and treats non-conforming file names as normal
|
||||
non-chunked files.
|
||||
|
||||
When using `norename` transactions, chunk names will additionally have a unique
|
||||
file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
|
||||
|
||||
|
||||
### Metadata
|
||||
|
||||
@@ -170,6 +173,7 @@ for composite files. Meta objects carry the following fields:
|
||||
- `nchunks` - number of data chunks in file
|
||||
- `md5` - MD5 hashsum of composite file (if present)
|
||||
- `sha1` - SHA1 hashsum (if present)
|
||||
- `txn` - identifies current version of the file
|
||||
|
||||
There is no field for composite file name as it's simply equal to the name
|
||||
of meta object on the wrapped remote. Please refer to respective sections
|
||||
@@ -242,8 +246,8 @@ use modification time of the first data chunk.
|
||||
|
||||
### Migrations
|
||||
|
||||
The idiomatic way to migrate to a different chunk size, hash type or
|
||||
chunk naming scheme is to:
|
||||
The idiomatic way to migrate to a different chunk size, hash type, transaction
|
||||
style or chunk naming scheme is to:
|
||||
|
||||
- Collect all your chunked files under a directory and have your
|
||||
chunker remote point to it.
|
||||
@@ -303,6 +307,8 @@ Chunker included in rclone releases up to `v1.54` can sometimes fail to
|
||||
detect metadata produced by recent versions of rclone. We recommend users
|
||||
to keep rclone up-to-date to avoid data corruption.
|
||||
|
||||
Changing `transactions` is dangerous and requires explicit migration.
|
||||
|
||||
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/chunker/chunker.go then run make backenddocs" >}}
|
||||
### Standard Options
|
||||
|
||||
|
||||
Reference in New Issue
Block a user