mirror of
https://github.com/rclone/rclone.git
synced 2025-12-11 22:14:05 +01:00
sftp: add support for about and hashsum on windows server
Windows shells like cmd and powershell needs to use different quoting/escaping of strings and paths than the unix shell, and also absolute paths must be fixed by removing leading slash that the POSIX formatted paths have (e.g. /C:/Users does not work in shell, it must be converted to C:/Users). Tries to autodetect shell type (cmd, powershell, unix) on first use. Implemented default builtin powershell functions for hashsum and about when remote shell is powershell. See #5763 Fixes #5758
This commit is contained in:
albertony
@@ -28,6 +28,9 @@ Note that some SFTP servers will need the leading / - Synology is a
|
||||
good example of this. rsync.net, on the other hand, requires users to
|
||||
OMIT the leading /.
|
||||
|
||||
Note that by default rclone will try to execute shell commands on
|
||||
the server, see [shell access considerations](#shell-access-considerations).
|
||||
|
||||
## Configuration
|
||||
|
||||
Here is an example of making an SFTP configuration. First run
|
||||
@@ -244,6 +247,116 @@ And then at the end of the session
|
||||
|
||||
These commands can be used in scripts of course.
|
||||
|
||||
### Shell access
|
||||
|
||||
Some functionality of the SFTP backend relies on remote shell access,
|
||||
and the possibility to execute commands. This includes [checksum](#checksum),
|
||||
and in some cases also [about](#about-command). The shell commands that
|
||||
must be executed may be different on different type of shells, and also
|
||||
quoting/escaping of file path arguments containing special characters may
|
||||
be different. Rclone therefore needs to know what type of shell it is,
|
||||
and if shell access is available at all.
|
||||
|
||||
Most servers run on some version of Unix, and then a basic Unix shell can
|
||||
be assumed, without further distinction. Windows 10, Server 2019, and later
|
||||
can also run a SSH server, which is a port of OpenSSH (see official
|
||||
[installation guide](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse)). On a Windows server the shell handling is different: Although it can also
|
||||
be set up to use a Unix type shell, e.g. Cygwin bash, the default is to
|
||||
use Windows Command Prompt (cmd.exe), and PowerShell is a recommended
|
||||
alternative. All of these have bahave differently, which rclone must handle.
|
||||
|
||||
Rclone tries to auto-detect what type of shell is used on the server,
|
||||
first time you access the SFTP remote. If a remote shell session is
|
||||
successfully created, it will look for indications that it is CMD or
|
||||
PowerShell, with fall-back to Unix if not something else is detected.
|
||||
If unable to even create a remote shell session, then shell command
|
||||
execution will be disabled entirely. The result is stored in the SFTP
|
||||
remote configuration, in option `shell_type`, so that the auto-detection
|
||||
only have to be performed once. If you manually set a value for this
|
||||
option before first run, the auto-detection will be skipped, and if
|
||||
you set a different value later this will override any existing.
|
||||
Value `none` can be set to avoid any attempts at executing shell
|
||||
commands, e.g. if this is not allowed on the server.
|
||||
|
||||
When the server is [rclone serve sftp](/commands/rclone_serve_sftp/),
|
||||
the rclone SFTP remote will detect this as a Unix type shell - even
|
||||
if it is running on Windows. This server does not actually have a shell,
|
||||
but it accepts input commands matching the specific ones that the
|
||||
SFTP backend relies on for Unix shells, e.g. `md5sum` and `df`. Also
|
||||
it handles the string escape rules used for Unix shell. Treating it
|
||||
as a Unix type shell from a SFTP remote will therefore always be
|
||||
correct, and support all features.
|
||||
|
||||
#### Shell access considerations
|
||||
|
||||
The shell type auto-detection logic, described above, means that
|
||||
by default rclone will try to run a shell command the first time
|
||||
a new sftp remote is accessed. If you configure a sftp remote
|
||||
without a config file, e.g. an [on the fly](/docs/#backend-path-to-dir])
|
||||
remote, rclone will have nowhere to store the result, and it
|
||||
will re-run the command on every access. To avoid this you should
|
||||
explicitely set the `shell_type` option to the correct value,
|
||||
or to `none` if you want to prevent rclone from executing any
|
||||
remote shell commands.
|
||||
|
||||
It is also important to note that, since the shell type decides
|
||||
how quoting and escaping of file paths used as command-line arguments
|
||||
are performed, configuring the wrong shell type may leave you exposed
|
||||
to command injection exploits. Make sure to confirm the auto-detected
|
||||
shell type, or explicitely set the shell type you know is correct,
|
||||
or disable shell access until you know.
|
||||
|
||||
### Checksum
|
||||
|
||||
SFTP does not natively support checksums (file hash), but rclone
|
||||
is able to use checksumming if the same login has shell access,
|
||||
and can execute remote commands. If there is a command that can
|
||||
calculate compatible checksums on the remote system, Rclone can
|
||||
then be configured to execute this whenever a checksum is needed,
|
||||
and read back the results. Currently MD5 and SHA-1 are supported.
|
||||
|
||||
Normally this requires an external utility being available on
|
||||
the server. By default rclone will try commands `md5sum`, `md5`
|
||||
and `rclone md5sum` for MD5 checksums, and the first one found usable
|
||||
will be picked. Same with `sha1sum`, `sha1` and `rclone sha1sum`
|
||||
commands for SHA-1 checksums. These utilities normally need to
|
||||
be in the remote's PATH to be found.
|
||||
|
||||
In some cases the shell itself is capable of calculating checksums.
|
||||
PowerShell is an example of such a shell. If rclone detects that the
|
||||
remote shell is PowerShell, which means it most probably is a
|
||||
Windows OpenSSH server, rclone will use a predefined script block
|
||||
to produce the checksums when no external checksum commands are found
|
||||
(see [shell access](#shell-access)). This assumes PowerShell version
|
||||
4.0 or newer.
|
||||
|
||||
The options `md5sum_command` and `sha1_command` can be used to customize
|
||||
the command to be executed for calculation of checksums. You can for
|
||||
example set a specific path to where md5sum and sha1sum executables
|
||||
are located, or use them to specify some other tools that print checksums
|
||||
in compatible format. The value can include command-line arguments,
|
||||
or even shell script blocks as with PowerShell. Rclone has subcommands
|
||||
[md5sum](/commands/rclone_md5sum/) and [sha1sum](/commands/rclone_sha1sum/)
|
||||
that use compatible format, which means if you have an rclone executable
|
||||
on the server it can be used. As mentioned above, they will be automatically
|
||||
picked up if found in PATH, but if not you can set something like
|
||||
`/path/to/rclone md5sum` as the value of option `md5sum_command` to
|
||||
make sure a specific executable is used.
|
||||
|
||||
Remote checksumming is recommended and enabled by default. First time
|
||||
rclone is using a SFTP remote, if options `md5sum_command` or `sha1_command`
|
||||
are not set, it will check if any of the default commands for each of them,
|
||||
as described above, can be used. The result will be saved in the remote
|
||||
configuration, so next time it will use the same. Value `none`
|
||||
will be set if none of the default commands could be used for a specific
|
||||
algorithm, and this algorithm will not be supported by the remote.
|
||||
|
||||
Disabling the checksumming may be required if you are connecting to SFTP servers
|
||||
which are not under your control, and to which the execution of remote shell
|
||||
commands is prohibited. Set the configuration option `disable_hashcheck`
|
||||
to `true` to disable checksumming entirely, or set `shell_type` to `none`
|
||||
to disable all functionality based on remote shell command execution.
|
||||
|
||||
### Modified time
|
||||
|
||||
Modified times are stored on the server to 1 second precision.
|
||||
@@ -255,6 +368,20 @@ upload (for example, certain configurations of ProFTPd with mod_sftp). If you
|
||||
are using one of these servers, you can set the option `set_modtime = false` in
|
||||
your RClone backend configuration to disable this behaviour.
|
||||
|
||||
### About command
|
||||
|
||||
SFTP supports the [about](/commands/rclone_about/) command if the
|
||||
same login has access to a Unix shell, where the `df` command is
|
||||
available (e.g. in the remote's PATH). `about` will return the
|
||||
total space, free space, and used space on the remote for the disk
|
||||
of the specified path on the remote or, if not set, the disk of
|
||||
the root on the remote. `about` will fail if it does not have
|
||||
shell access or if `df` is not found.
|
||||
|
||||
If the server shell is PowerShell, probably with a Windows OpenSSH
|
||||
server, rclone supports `about` using a built-in shell command
|
||||
(see [shell access](#shell-access)).
|
||||
|
||||
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/sftp/sftp.go then run make backenddocs" >}}
|
||||
### Standard options
|
||||
|
||||
@@ -637,25 +764,9 @@ Properties:
|
||||
|
||||
## Limitations
|
||||
|
||||
SFTP supports checksums if the same login has shell access and `md5sum`
|
||||
or `sha1sum` as well as `echo` are in the remote's PATH.
|
||||
This remote checksumming (file hashing) is recommended and enabled by default.
|
||||
Disabling the checksumming may be required if you are connecting to SFTP servers
|
||||
which are not under your control, and to which the execution of remote commands
|
||||
is prohibited. Set the configuration option `disable_hashcheck` to `true` to
|
||||
disable checksumming.
|
||||
|
||||
SFTP also supports `about` if the same login has shell
|
||||
access and `df` are in the remote's PATH. `about` will
|
||||
return the total space, free space, and used space on the remote
|
||||
for the disk of the specified path on the remote or, if not set,
|
||||
the disk of the root on the remote.
|
||||
`about` will fail if it does not have shell
|
||||
access or if `df` is not in the remote's PATH.
|
||||
|
||||
Note that some SFTP servers (e.g. Synology) the paths are different for
|
||||
SSH and SFTP so the hashes can't be calculated properly. For them
|
||||
using `disable_hashcheck` is a good idea.
|
||||
On some SFTP servers (e.g. Synology) the paths are different
|
||||
for SSH and SFTP so the hashes can't be calculated properly.
|
||||
For them using `disable_hashcheck` is a good idea.
|
||||
|
||||
The only ssh agent supported under Windows is Putty's pageant.
|
||||
|
||||
@@ -670,11 +781,10 @@ SFTP isn't supported under plan9 until [this
|
||||
issue](https://github.com/pkg/sftp/issues/156) is fixed.
|
||||
|
||||
Note that since SFTP isn't HTTP based the following flags don't work
|
||||
with it: `--dump-headers`, `--dump-bodies`, `--dump-auth`
|
||||
with it: `--dump-headers`, `--dump-bodies`, `--dump-auth`.
|
||||
|
||||
Note that `--timeout` and `--contimeout` are both supported.
|
||||
|
||||
|
||||
## rsync.net {#rsync-net}
|
||||
|
||||
rsync.net is supported through the SFTP backend.
|
||||
|
||||
Reference in New Issue
Block a user