Skip to main content

​
How do I upload/download to/from my instance?

You can use the CLI copy command to copy from/to directories on a remote instance and your local machine, or to copy data between two remote instances. You can use the copy buttons in the GUI to copy data between two remote instances. The copy command uses rsync and is generally fast and efficient, subject to single link upload/download constraints. The copy command supports multiple location formats:
  • [instance_id:]path - Legacy format (still supported)
  • C.instance_id:path - Container copy format
  • cloud_service:path - Cloud service format
  • cloud_service.cloud_service_id:path - Cloud service with ID
  • local:path - Explicit local path
Examples: 
vastai copy 6003036:/workspace/ 6003038:/workspace/
vastai copy C.11824:/data/test local:data/test
vastai copy local:data/test C.11824:/data/test
vastai copy drive:/folder/file.txt C.6003036:/workspace/
vastai copy s3.101:/data/ C.6003036:/workspace/
The first example copy syncs all files from the absolute directory ‘/workspace’ on instance 6003036 to the directory ‘/workspace’ on instance 6003038. The second example copy syncs files from container 11824 to the local machine using structured syntax. The third example copy syncs files from local to container 11824 using structured syntax. The fourth example copy syncs files from Google Drive to an instance. The fifth example copy syncs files from S3 bucket with id 101 to an instance. Important: You should not copy to /root or / as a destination directory, as this can mess up the permissions on your instance ssh folder, breaking future copy operations (as they use ssh authentication). You can see more information about constraints here: https://vast.ai/docs/gpu-instances/data-movement#constraints Currently, one endpoint of the copy must involve a vast instance with open ports. For a remote->local copy or local->remote copy, the remote instance must be on a machine with open ports (although the instance itself does not need open ports), and the remote instance can be stopped/inactive. For instances on machines WITHOUT open ports, copy to/from local is not available, but you can still copy to a 2nd vast instance with open ports. For a remote->remote copy (copy between 2 instances), the src can be stopped and does not need open ports, but the dst must be a running instance with open ports. It is not sufficient that the instance is on a machine with open ports, the instance itself must have been created with open port mappings. If the instance is created with the direct connect option (for jupyter or ssh launch modes), the instance will have at least one open port. Otherwise, for proxy or entrypoint instance types, you can get open ports using the -p option to reserve a port in the instance configuration under run options (and you must also then pick a machine with open ports). If your data is already stored in the cloud (S3, gdrive, etc) then you should naturally use the appropriate linux CLI or commands to download and upload data directly. This generally will be one the fastest methods for moving large quantities of data, as it can fully saturate a large number of download links. If you are using multiple instances with significant data movement requirements you will want to use high bandwidth cloud storage and avoid any single machine bottlenecks. If you launched a Jupyter notebook instance, you can use its upload feature, but this has a file size limit and can be slow. You can also use standard Linux tools like scp, ftp, rclone, or rsync to move data. For moving code and smaller files scp is fast enough and convenient. However, be warned that the default ssh connection uses a proxy and can be slow for large transfers.

​
How do I upload/download to/from my instance - using scp?

If you launched an ssh instance, you can copy files using scp. The default ssh connection uses a proxy and thus can be slow (in terms of latency and bandwidth). Thus we recommend only using scp over the default ssh connection for smaller transfers (less than 1 GB). For larger inbound transfers, a direct connection is recommended. Downloading from a cloud data store using wget or curl can have much higher performance. The relevant scp command syntax is: 
scp -p PORT LOCAL_FILE root@IPADDR:/REMOTEDIR
The PORT and IPADDR fields must match those from the ssh command. The “Connect” button on the instance will give you these fields in the form: 
ssh -p PORT root@IPADDR -L 8080:localhost:8080
For example, if Connect gives you this: 
ssh -p 7417 root@52.204.230.7 -L 8080:localhost:8080
You could use scp to upload a local file called “myfile.tar.gz” to a remote folder called “mydir” like so: 
scp -p 7417 myfile.tar.gz root@52.204.230.7:/path/to/mydir

​
I’m getting a ConnectionResetError downloading files?

This seems to be due to bugs in the urllib3 and or requests libraries used by many Python packages. We recommend using wget to download large files - it is quite robust and recovers from errors gracefully.

​
How can I download a Kaggle dataset using wget?

First, you need to get the raw https link. Using the Chrome browser, on the Kaggle website go to the relevant dataset or competition page and start downloading the file you want. Then cancel the download and press Ctrl+J to bring up the Chrome Downloads page. At the top is the most recent download with a name and a link under it. Right-click on the URL link and use “copy link address”. Then you can use wget with that URL as follows: 
wget 'URL' --no-check-certificate -O FILENAME
Notice the URL needs to be wrapped in ’ ’ single quotes.
⌘I