Cloud Computing at Aalto University

How to transfer files between a local machine and a virtual machine

In this article, we will show you examples of four easy ways to transfer files between your local machine (PC, laptop) and a virtual machine.

There is a common need for our researchers to securely transfer small or large files to virtual machines for various purposes. They also need to download some result files or data to their local machines. We are referring to Linux virtual machines in this post unless we explicitly mention Windows virtual machines as well. 

There are no requirements for retaining the permissions on the files being transferred. In this post, we will not go through performance considerations for each tool/method.  

We decided to write down four ways to achieve this goal.  

  1. SCP 
  2. SFTP 
  3. RSYNC 
  4. Azure File Share 

The first three methods can be used to access any virtual machines that you have ssh connection. If you are using Azure virtual machines and login with Azure Active Directory, you will get more instructions on how to connect. The last method is specific to Microsoft Azure and requires a bit of setup by Azure administrator as well as installing some software.  

SCP 

SCP is a common tool for secure copy files between a local and remote machine. It uses the same authentication and security as SSH. You need a SCP client which is included in Bash shell of most Linux distributions and Mac computers and PowerShell.  

Examples 

If you are using username and password for SSH authentication, you will be prompted to enter your password. However, if you are using SSH key pair authentication, SSH will authenticate you using your private key and there is no need to enter your password (best practice). If you created your key pairs with a passphrase, you will be prompted to enter the passphrase.  

From local to remote  $ scp file username@host-ip:target-directory 
From remote to local  $ scp username@host-ip:target-directory/file /local-directory 

If you login to your Azure virtual machine using Azure Active Directory (AD), you need to create a temporary config file to use with SCP (thanks to Nigel Sim post). You are already familiar with login with AD using the command below: 

$ az ssh vm --ip 1.2.3.4 

or

$ az ssh vm --resource-group myResourceGroup --name myVM

The config file can be generated for one specific virtual machine using these commands:

Using VM IP address $ az ssh config --ip 1.2.3.4 --file ./sshconfig
Using Resource Group/VM name

$ az ssh config --resource-group myResourceGroup --name myVm --file ./sshconfig

Or it can be generated as a generic config file to use with any host: 

$ az ssh config --ip \* --file ./sshconfig 

This will generate Azure certificates and SSH keys required for connection. These config files can be used by any client that supports OpenSSH configs and certificates. Please refer to Microsoft official documentation for more details on this topic. 

Important note: The config file contains sensitive information and should be deleted when not needed. At the time of writing this post, there is a time limit for validity of the certificates and SSH keys.  

Now that the configs are ready, we can use SCP using the commands below: 

From local to remote  $ scp -F ./sshconfig file 1.2.3.4:~/  
From remote to local  $ scp -F ./sshconfig 1.2.3.4:~/file /local-directory 

Example for copying a folder from local to remote:

$ scp -F ./sshconfig -r folder-to-be-copied 1.2.3.4:~/

SFTP 

SFTP is another easy-to-use tool for transferring files over SSH and it replaces FTP. Like SCP, SFTP client is also included in Bash shell in most Linux and Mac computers as well as PowerShell. First, you need to connect to your remote machine using the command below: 

$ stfp username@host-ip 

Or in the case of Azure AD login: 

$ sftp –F ./sshconfig 1.2.3.4 

After successful authentication, you will be inside a sftp session. You can use ‘help’ command to learn how to move files from local machine to remote and vice versa. Learn more about those commands (‘put’ and ‘get’) here.  

$ sftp>help

RSYNC 

RSYNC is a robust tool for transferring large files securely. It uses delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Additionally, it handles file transfer interruptions elegantly.  

Examples

From local to remote  $ rsync -avP file username@host-ip:~/ 
From remote to local  $ rsync -avP username@host-ip:~/file /local-directory 

And if you are using Azure AD authentication and have your SSH config file created as instructed above, use the commands below:

From local to remote  $ rsync -e 'ssh -F ./sshconfig' -avP file 1.2.3.4:~/target-directory 
From remote to local  $ rsync -e 'ssh -F ./sshconfig' -avP 1.2.3.4:~/file /local-directory 

Azure File Share 

Azure Files is a fully managed service for file shares in the cloud which are accessible via SMB (Server Message Block) protocol or NFS (Network File System) protocol. Azure file shares can be mounted in Windows, Linux, and macOS. You can mount the file shares to virtual machines and local machines as illustrated in the image below. 

Azure File Shares

For Azure Admins 

There are many well-written tutorials on how to set up Azure Storage account and create a file share. Here are official instructions on the Microsoft website. 

When your file share(s) are created, follow instructions for mounting the shares in Windows or Linux from the links provided. Please note that mounting the shares on local machines and virtual machines is similar with some consideration:  

  • Azure Storage Account and the virtual machine need to be in the same Azure region. Otherwise, the OS must support the encryption functionality of SMB 3.0.  

  • There is a requirement for SMB 3.0 to use TCP port 445 to establish a secure connection. If this port is closed in your network, you can download free Azure Storage Explorer on your machine to upload and download files in file share. Alternatively, if you have access rights, you can use Azure portal and storage account to do file transfer operations from browser. 

Once the file shares are mounted, they can be treated like normal drives on the machine. You can start transferring files to the mounted drive.  


Post author

Mohammad Bakharzy

Cloud Solutions Consultant

IT Operations, Cloud and Apps

 

Learn more and contact us:

Cloud Computing at Aalto

  • Published:
  • Updated:
Share
URL copied!

Show other posts from this blog

virtual computer lab
Published:

Virtual Labs for Deep Learning with Audio Course

A review of Azure Lab Services in teaching the "Deep Learning with Audio" course at Aalto
Jupyter-Notebooks-Azure-machine-learning logos
Published:

Run Jupyter Notebooks on Azure

How to run Jupyter Notebook and JupyterLab on Microsoft Azure cloud computing environment.
Azure Lab Services
Published:

Azure Lab Service for Teaching

Learn how Azure Lab Services helps teachers and students