johanels/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-12-11 22:14:05 +01:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity

operations: reopen downloads on error when using check --download and cat

Browse Source 
Before this change, some parts of operations called the Open method on
objects directly, and some called NewReOpen to make an object which
can re-open itself on errors.

This adds a new function operations.Open which should be called
instead of fs.Object.Open to open a reliable stream of data and
changes all call sites to use that.

This means `rclone check --download` and `rclone cat` will re-open
files on failures.

See: https://forum.rclone.org/t/does-rclone-support-retries-for-check-when-using-download-flag/38641
This commit is contained in:
Nick Craig-Wood
2023-06-01 12:54:19 +01:00
parent 279d9ecc56
commit 1f9c962183
4 changed files with 32 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
fs/operations/reopen.go
View File

@@ -29,12 +29,14 @@ var (
errorTooManyTries = errors.New("failed to reopen: too many retries")
)
// NewReOpen makes a handle which will reopen itself and seek to where it was on errors
// NewReOpen makes a handle which will reopen itself and seek to where
// it was on errors up to maxTries times.
//
// If hashOption is set this will be applied when reading from the start.
// If an fs.HashesOption is set this will be applied when reading from
// the start.
//
// If rangeOption is set then this will applied when reading from the
// start, and updated on retries.
// If an fs.RangeOption is set then this will applied when reading from
// the start, and updated on retries.
func NewReOpen(ctx context.Context, src fs.Object, maxTries int, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
h := &ReOpen{
ctx: ctx,
@@ -51,6 +53,24 @@ func NewReOpen(ctx context.Context, src fs.Object, maxTries int, options ...fs.O
return h, nil
}
// Open makes a handle which will reopen itself and seek to where it
// was on errors.
//
// If an fs.HashesOption is set this will be applied when reading from
// the start.
//
// If an fs.RangeOption is set then this will applied when reading from
// the start, and updated on retries.
//
// It will obey LowLevelRetries in the ctx as the maximum number of
// tries.
//
// Use this instead of calling the Open method on fs.Objects
func Open(ctx context.Context, src fs.Object, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
maxTries := fs.GetConfig(ctx).LowLevelRetries
return NewReOpen(ctx, src, maxTries, options...)
}
// open the underlying handle - call with lock held
//
// we don't retry here as the Open() call will itself have low level retries