chunker: partially implement no-rename transactions (#4675)

Some storage providers e.g. S3 don't have an efficient rename operation.
Before this change, when chunker finished an upload, the server-side copy
and delete operations that renamed temporary chunks to their final names
could take a significant amount of time.
This PR records transaction identifier (versioning) in the metadata of
chunker composite objects striving to remove the need for rename
operations on such backends.
This approach will be triggered be the new "transactions" configuration
option, which can be "rename" (the default) or "norename".
We implement the new approach for uploads (Put operations).
The chunker Move operation still uses the rename operation of
underlying backend. Filling this gap is left for a later PR.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
This commit is contained in:
Maxwell Calman
2020-10-30 15:30:04 -05:00
committed by Ivan Andreev
parent b029fb591f
commit 9cc8ff4dd4
4 changed files with 317 additions and 53 deletions

View File

@@ -151,6 +151,9 @@ Note that `list` assembles composite directory entries only when chunk names
match the configured format and treats non-conforming file names as normal
non-chunked files.
When using `norename` transactions, chunk names will additionally have a unique
file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
### Metadata
@@ -170,6 +173,7 @@ for composite files. Meta objects carry the following fields:
- `nchunks` - number of data chunks in file
- `md5` - MD5 hashsum of composite file (if present)
- `sha1` - SHA1 hashsum (if present)
- `txn` - identifies current version of the file
There is no field for composite file name as it's simply equal to the name
of meta object on the wrapped remote. Please refer to respective sections
@@ -242,8 +246,8 @@ use modification time of the first data chunk.
### Migrations
The idiomatic way to migrate to a different chunk size, hash type or
chunk naming scheme is to:
The idiomatic way to migrate to a different chunk size, hash type, transaction
style or chunk naming scheme is to:
- Collect all your chunked files under a directory and have your
chunker remote point to it.
@@ -303,6 +307,8 @@ Chunker included in rclone releases up to `v1.54` can sometimes fail to
detect metadata produced by recent versions of rclone. We recommend users
to keep rclone up-to-date to avoid data corruption.
Changing `transactions` is dangerous and requires explicit migration.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/chunker/chunker.go then run make backenddocs" >}}
### Standard Options