Data Transfer with SFTP

This page describes how to transfer data to and from the HPC and Spear filesystems (GPFS) using SFTP.

Choosing a transfer option

There are two basic ways to move data to and from our GPFS storage system:

  1. Secure File Transfer Protocal (SFTP) - Best when your data transfer is relatively small and will take only a few minutes, or your connected on-campus via a realiable high-speed network.  This method is quick and conveneint, but less reliable for long-running transfers.
  2. Globus - Best when your data transfer will take a long time (at least several hours) or when the connection may be peroidically interrupted during transfer.  This method requires a little setup ahead of time, but is robust and appropriate for any data transfer task.  Read more about this option.

For the Research Archival storage system, you can access the data to and from our systems via Globus.

Using SFTP with Cyberduck (Mac/Windows)

CyberDuck is a popular open source SFTP client for Windows and Mac.

Download and install Cyberduck from the Cyberduck website.

Open the Cyberduck application.

Click the "Open Connection" button on the toolbar.

Select "SFTP (SSH File Transfer Protocol)" from the drop-down.

Enter in the Server box.

Enter your RCC credentials, and click Connect.

If the application prompts you to allow the fingerprint, click "Allow"

Your files on the server will appear in the CyberDuck window.  You can now drag-and-drop files to and from the window to upload/download files.

Using SFTP from Natilus File Manager (Linux)

Most popular Linux distributions with graphical environments (Ubuntu, RHEL, CentOS, etc) include the Natilus File Manager.  This means that you can connect directly from your system to our servers without installing additional software.

Open up your file manager, and locate the Connect to Server link on the left side of the file browser.

In the dialog box, enter s to connect to the HPC filesystem (gpfs).

Enter your RCC credentials when prompted.

Once logged in, the file system will appear as a location on the left-hand side of the Natilus File manager, the same as any other file or folder on your computer.  You can drag and drop, copy and paste, and interact with the files on the server the same way that you do with files on your computer.  Press the eject button to disconnect.

Using SFTP on the Command-Line (Mac/Linux)

Using SFTP from the command-line, you can make use of the sftp command, which comes bundled with most Linux distributions and Mac OS X.  You can also use the scp command on Linux.  Examples:

# Connect to the HPC Filesystem (gpfs)
$ sftp's password: 
Connected to

# List files on the server
sftp> ls

# Copy local file (/tmp/test.txt) to the server
sftp> put /tmp/test.txt
/tmp/test.txt 100%   88KB  88.5KB/s   00:00    

# Copy remote file on server to computer (downloads to current working directory)
sftp> get test.txt

# Disconnect
sftp> exit

# Use scp to copy a file immediately to home directory on HPC (useful for scripting transfers)
$ scp /tmp/test.txt

# Use scp to copy an entire directory to HPC with -r option
$ scp -r ~/Documents/my_research_data

Learn more about the SFTP command with this guide from DigitalOcean.