Chapter 7
Secure File Transfer
The Secure File Transfer mechanism consists of three programs — the client program SCP2, which includes an embedded SFTP server for local file access, and SFTP-SERVER2 and
SCP-SERVER1, which run on the remote system to access the file. SCP2 communicates with SSH2 for authentication and data transport (which includes encryption) to remote systems,
SFTP-SERVER2 communicates with SSHD2 for data transport.
The following diagram illustrates the relationship among the client and server portions of an SCP2 file transfer: see graphic
SCP file transfers are different from FTP file transfers. With FTP a file can be transferred as ASCII, BINARY, RECORD, or in VMS Plus mode (if Compaq TCP/IP Services for OpenVMS is in use). In SCP there is one specified format — BINARY. Also, the defined syntax for a file specification is UNIX syntax. Due to these restrictions, files that are transferred from dissimilar systems may or may not be useful. Process Software has used methods available in the protocol to attempt to improve the chances that files will be useful upon transfer. The SSH File Transfer Protocol is an evolving specification, and some implementations may not support all options available in the protocol, or worse, not tolerate some optional parts of later versions of the protocol.
Process Software has used the defined extensions in the protocol to transfer information about the VMS file header characteristics such that when a file is transferred between two VMS systems running MultiNet v4.4, TCPware v5.6, and/or SSH for OpenVMS, the file header information will also be transferred and the file will have the same format on the destination system as it had on the source system. Also, when a file is transferred to a non-VMS system, a method has been provided to translate those files that can be translated into a format that will be usable on the remote system. Files that are transferred from non-VMS systems are stored as stream files on the VMS system, which provides compatibility for text files from those systems.
The SCP-SERVER1 program is used when a system with OpenSSH initiates an SCP command. OpenSSH uses RCP over SSH2 instead of the SFTP protocol. SCP-SERVER1 will always translate VMS text files (if possible) when copying a file from VMS. Translated VMS text files may have some trailing nulls at the end of them, due to the RCP protocol not being able to tolerate a file that comes up short of the reported size. SCP-SERVER1 (and SFTP-SERVER2) use sophisticated methods to estimate the amount of user data in the file to minimize this. On ODS-5 disks the estimation routine uses the file size hint if it is valid. On ODS-2 disks (and ODS-5 without a valid size hint), the size of the file and file characteristics are used to estimate the amount of user data. The method provides as accurate an estimate as possible without actually reading the file and never underestimates the amount of data in the file. Underestimating would cause significant problems as the programs use the size of the file to determine how much data to expect.
SCP2 [qualifiers] [[user@]host[#port]::]file [[user@]host[#port]::]file
Note! The source and destination file specification must be quoted if they contain a user specification or a non-VMS file specification.
Note! /VMS and /TRANSLATE_VMS are mutually exclusive
The source and destination strings are changed to lowercase unless they are enclosed in quotes, in which case they are left the same. File specification must be in UNIX format for remote systems, unless the remote system is running TCPware 5.6, MultiNet v4.4, or SSH for OpenVMS, and
/VMS or /TRANSLATE_VMS (source files only) are used. UNIX format file specifications need to be enclosed in quotes (") if they contain the / character to prevent the DCL parsing routines from interpreting the string as a qualifier.
/BATCH
Starts SSH2 in BATCH mode. When SSH2 is running in BATCH mode it does not prompt for a password, so user authorization is accomplished by Public-Key authentication.
/BUFFER_SIZE=integer
Number of bytes of data to transfer in a buffer. Default is 7500.
/CIPHER=(cipher,...,cipher-n)
Lets you select which SSH2 cipher to use.
/COMPRESS
Enables SSH2 data compression. This can be beneficial for large file transfers over slow links. The compression level is set by the client configuration file for SSH2.
/CONCURRENT_REQUEST=integer
Number of concurrent read requests to post to the source file. Default is 4.
/DEBUG
Enables debugging messages for SCP2 and SSH2. Higher numbers get more messages. The legal values are between 0 (none) and 99. Debugging for SFTP-SERVER2 is enabled via the MULTINET_SSH_SFTP_SERVER_DEBUG logical.
/DIRECTORY
Informs SCP2 that the target specification should be a directory that the source file(s) will be put in. This qualifier is necessary when using wildcards in the source file specification, or /RECURSIVE.
/HELP
Displays command qualifier list and parameter format.
/IDENTITY_FILE=file
Specifies the identity file that SSH2 should use for Public-Key authentication.
/PORT=number
Specifies the port that SSH2 uses on the remote system. Note that if both the source and destination files are remote, this value is applied to both. If SSH2 is available on different ports on the two systems, then the #port method must be used.
/PRESERVE
Sets the Protection, Owner (UIC), and Modification dates on the target file to match that of the source file. /PRESERVE is not very useful when the target machine is a VMS system as VMS does not provide runtime library calls for setting the file attributes (owner, protection) and timestamps. Note that the VMS modification date (not the creation date) is propagated to the remote system. When files are copied between two VMS systems and /VMS is used /PRESERVE is implied and the process of transferring VMS attributes preserves the information about the protection, dates, and file characteristics. The file access time is not adjusted for the difference between local time and GMT.
/NOPROGRESS
SCP2, by default, updates a progress line at regular intervals when it is run interactively to show how much of the file has been transferred. This qualifier disables the progress line.
/QUIET
Disables warning messages. Note that it does not disable warning messages from SFTP-SERVER2, which return on the error channel.
/RECURSIVE
Copies all of the files in the specified directory tree. Note that the top level directory on the local system is not created on the remote system. When /VMS is used, all versions of the files are copied. When /VMS is not used, only the most recent version is copied.
/REMOVE
Deletes the source files after they have been copied to the remote system.
/TRANSLATE_VMS
Translates VMS text files in the copying process to byte streams separated by linefeeds because the defined data transfer format for SCP2 is a binary stream of bytes.
/TRANSLATE_VMS is only applicable to the source specification. If a remote source file is specified, then that system must be running MultiNet v4.4, TCPware 5.6, or SSH for OpenVMS. If /TRANSLATE_VMS is specified with no value, then VARIABLE, FIXED, and VFC (Variable, Fixed Control) files are translated to stream linefeed files. If the value is NONE, no files are translated. VARIABLE, FIXED, and VFC can be combined in any manner. The SFTP-SERVER2 process also uses the value of the logical MULTINET_SFTP_TRANSLATE_VMS_FILE_TYPES to determine which files should be translated. This is a bit mask with bit 0 (1) = FIXED, bit 1 (2) = VARIABLE, and bit 2 (4) = VFC. These values can be combined into a number between 0 and 7 to control which files are translated.
Note! Due to the structure of the programs, the SCP2 program uses this logical if the
/TRANSLATE_VMS qualifier has not been specified.
/VERBOSE
Displays debugging messages that allow the user to see what command was used to start up SSH and other basic debugging information. Note that debugging information can interfere with the normal display of the progress line. Equivalent to /DEBUG=2.
/VERSION
Displays the version of the base SCP2 code.
/VMS
Transfers VMS file information similar to that transferred in OVMS mode in FTP such that VMS file structure can be preserved. All of the information transferred in FTP OVMS mode is transferred along with the file creation date and protection. If the file is a contiguous file, and it is not possible to create the file contiguously, and the logical MULTINET_SFTP_FALLBACK_TO_CBT has the value of TRUE, YES, or 1, SFTP-SERVER2 attempts to create the file Contiguous, Best Try. VMS mode is only available with SCP2 provided in MultiNet v4.4 and TCPware 5.6, and SSH for OpenVMS. /VMS also modifies the usual sequence of operations that SCP2 does such that a new version of the file is created if there are existing versions. Without /VMS, the most recent version of the target file is deleted (if it exists) before the new file is written. This is to accommodate systems that do not handle multiple versions of files.
The logical name MULTINET_SCP2_VMS_MODE_BY_DEFAULT can be defined to TRUE, YES, or 1 to specify that /VMS should be the default unless /NOVMS or /TRANSLATE_VMS are specified. /VMS and /TRANSLATE_VMS can not be used on the same command line. If /VMS is not specified, but the logical is set to enable it by default, a /TRANSLATE_VMS on the command line will take precedence.
Note that even though SCP2 & SFTP-SERVER2 pass the request for VMS file transfers or to translate a VMS file in a manner that is consistent with the protocol specification, other implementations may not handle this information well. Since there is no error response present at that point in the protocol, the program hangs. To prevent it from hanging forever, the logical MULTINET_SCP2_CONNECT_TIMEOUT is checked to see how long SCP2 should wait for a response when establishing the connection. The format for this logical is a VMS delta time. The default value is 2 minutes. If SCP2 times out before a connection is established with SFTP-SERVER2 and /VMS or /TRANSLATE_VMS were specified, a warning message is displayed and the initialization is tried again without the request for VMS information (or /TRANSLATE_VMS). This retry is also subject to the timeout, and if the timeout happens again, then SCP2 exits. This helps for implementations that ignore the initialization message when information they do not recognize is present; implementations that abort will cause SCP2 to exit immediately.
Logicals
The following logicals apply to both SCP2 and SFTP-SERVER:
MULTINET_SFTP_FALLBACK_TO_CBT MULTINET_SFTP_TRANSLATE_VMS_FILE_TYPES
MULTINET_SFTP_RETURN_ALQ
MULTINET_SFTP_FALLBACK_TO_CBT
When defined to TRUE, YES, or 1 and a VMS file transfer is being performed, this logical creates a Contiguous file if that file has Contiguous characteristics. The file will be created as Contiguous Best Try if there is insufficient space to create it as Contiguous.
MULTINET_SFTP_TRANSLATE_VMS_FILE_TYPES
This is a bit mask that determines which VMS file types should be translated when not operating in VMS mode.
• Bit 0 (1) = FIXED
• Bit 1 (2) = VARIABLE
• Bit 2 (4) = VFC
The values are:
• 0 (zero) = NONE
• 7 = ALL
Note that this logical affects SCP2 as well as the server, as SCP2 has the server built into it for handling local file access.
MULTINET_SCP2_CONNECT_TIMEOUT
This logical defines a number specifying how long SCP2 should wait for a response to the INITIALIZE command from the server program. This is a VMS delta time number. The default is 2 minutes.
MULTINET_SCP2_VMS_MODE_BY_DEFAULT
When defined to TRUE, YES, or 1, this logical chooses the /VMS qualifier if /TRANSLATE_VMS or /NOVMS has not been specified.
MULTINET_SFTP_RETURN_ALQ
When defined to TRUE, YES, or 1 and files are being transferred in VMS mode, this logical includes the Allocation Quantity for the file in the file header information. This is disabled by default because copying a small file from a disk with a large cluster size to a disk with a small cluster size causes the file to be allocated with more space than necessary. You have the option of retaining the allocated size of a file if it was allocated the space for a reason. Some combinations of file characteristics require that the Allocation Quantity be included in the file attributes; this is handled by SCP2/SFTP-SERVER2.
MULTINET_SSH_SCP_SERVER_DEBUG
Enables debugging messages for the SCP-SERVER1 image that provides service to SCP commands that use the RCP over SSH2 protocol (OpenSSH). When this is defined, the file
SCP-SERVER.LOG is created in the user’s login directory. These files are not purged. Larger values yield more debugging information.
MULTINET_SSH_SFTP_SERVER_DEBUG
Enables debugging messages for the SFTP-SERVER2 image that provides service to SCP2 commands that use the SFTP protocol. When this is defined, the file SFTP-SERVER.LOG is created in the user’s login directory. These files are not purged. Larger values yield more debugging information.
SSH2 can be used to set up port forwarding that can be used for FTP. This allows users to use the richness of the FTP command set to access files on a remote system and have their control and data information encrypted. The command format to set up the SSH port forwarding is:
$ ssh <remote_host_name>/local_forward=
("""ftp/<forwarded_port_number>:localhost:21""")
The usual SSH authentication mechanisms come into play, so there may be a request for a password and a terminal session is established to the remote host. As long as this terminal session is alive, other users on the local system can use FTP to access the remote system over an encrypted channel. The location of the quotes is important, as it is necessary to prevent DCL from interpreting the / in the local forwarding information as the start of a new qualifier, and SSH2 does not know or expect to find the ( ) around the forwarding information. Note that the "localhost" inside of the forwarding string is important, as it will make the connection to FTP on the remote system come from localhost, which will then allow FTP to open the data port.
When a user desires to use an encrypted FTP connection, the following OPEN command would be issued to the TCP/IP Services for OpenVMS FTP client:
OPEN LOCALHOST <forward_port_number>
Normal FTP authentication takes place and multiple FTP sessions may use a single forwarded port. The FTP protocol filter in SSH2 scans the FTP command stream for the FTP PORT and PASV commands and their replies, and makes substitutions in these commands and replies to use a secure data stream through the SSH2 session that has been set up.
To allow a single system to act as a gateway between two networks, add
/ALLOW_REMOTE_CONNECT to the SSH command that initiates the connection. This command will establish an encrypted FTP session with the remote host that the SSH connection is sent to.
