Vast.ai provides Linux docker instances. Your template (or CLI command) controls most of the parameters to the underlying docker create call, but most resource constraint options naturally must be configured automatically by our system.

We automatically assign CPU, RAM, shared mem, and related cgroup resources automatically in proportion to your instance's cost vs the total machine cost.

These launch modes attempt to 'inject' Jupyter/SSH setup in to your existing docker image. The image entrypoint is replaced, so if you are using Jupyter or SSH launch mode for a docker image that uses an entrypoint script, you typically want to take the entrypoint command and add it to the end of your onstart. Sometimes the injection scripts can cause obscure loading errors. If you run into such issues with a custom image it may be best to use the simpler args/entrypoint launch mode and setup ssh/jupyter yourself.

You can currently set 3 types of docker create/run options in the GUI and CLI:

environment variables: "-e JUPYTER_DIR=/ -e TEST=OK"

hostname: "-h billybob"

ports: "-p 8081:8081 -p 8082:8082/udp -p 70000:70000"

Use the -e docker syntax in the docker create/run options to set env variables. For example, to set the env variables TZC to UTC and TASKID to "TEST":

Any environment variables you set will be visible only to your onstart script (or your entrypoint for entrypoint launch mode). When using the SSH or Jupyter launch modes, your env variables will not be visible inside your SSH/tmux/Jupyter session by default. To make custom environment variables visible to the shell, you need to export them to /etc/environment.

Add something like the following to the end of your onstart to export any env variables containing an underscore '_':

Or to export all env variables:

Users can set their environment variables in their User Account settings, which will automatically inject these variables into every container they use.

Some special environment variables are used to signal to the interface:

OPEN_BUTTON_PORT: Set this to map the open button on the instance panel to a specific (external) port corresponding to the specified internal port.

For example:

Will map the open button to whatever external port maps to internal port 7860.

JUPYTER_PORT: Use this to control the jupyter button. Set this to your internal jupyter port and the UI will map the jupyter button to open jupyter on the corresponding IP in a new tab.

For example:

Will map the jupyter button to whatever external port maps to internal port 8081.

JUPYTER_TOKEN: Use this to control the jupyter button. Set this to your jupyter token and the UI will map the jupyter button to open jupyter using the corresponding JUPYTER_TOKEN in a new tab.

For example:

Will use TOKEN as a value of your jupyter Token.

DATA_DIRECTORY: This env variable is used as the default src or dst directory for data copy operations.

Our system also predefines some environment variables you can use:

CONTAINER_API_KEY: Per instance API key you can use to access some CLI commands from within the instance.

CONTAINER_ID: The unique ID of your instance.

DATA_DIRECTORY: Location on the instance to copy data to/from

GPU_COUNT: Number of GPU devices.

PUBLIC_IPADDR: The instance's public IP address.

SSH_PUBLIC_KEY: Your SSH public key from the account page.

PYTORCH_VERSION: The pytorch version (if applicable)

JUPYTER_TOKEN: The Jupyter access token.

JUPYTER_SERVER_ROOT: The root directory for Jupyter (can't navigate above this!)

JUPYTER_SERVER_URL: Configured jupyter server URL (usually https://0.0.0.0:8080/)

VAST_CONTAINERLABEL: Also the unique name/ID of your instance.

Port env variables:

VAST_TCP_PORT_22: The external public TCP port that maps to internal port 22 (ssh). VAST_TCP_PORT_8080: The external public TCP port that maps to internal port 22 (ssh).

For each internal TCP port request: VAST_TCP_PORT_X: The external public TCP port that maps to internal port X.

For each internal UDP port request: VAST_TCP_PORT_X: The external public UDP port that maps to internal port X.

You can also use ports 70000 and above for identity port mappings (see networking below).

Vast.ai docker instances have full internet access, but generally do not have unique IP addresses. Instances can have public open ports, but as IP addresses are shared across machines/instances the public external ports are partitioned somewhat randomly. In essence each docker instance gets a fraction of a public IP address based on a subset of ports. Each open internal port (such as 22 or 8080 etc) is mapped to a random external port on the machine's (usually shared) public IP address.

Selecting the ssh launch mode will open and use port 22 internal by default, whereas jupyter will open and use port 8080 (in addition to 22 for ssh). There are several ways to open additional application ports:

Note: there is currently a limit of 64 total open ports per container/instance.

Any EXPOSE commands in your docker image will be automatically mapped to port requests. You can also open custom ports for any docker image more dynamically using -p arguments in the docker create/run options box in the image config editor pop-up menu. To open ports 8081 (tcp) and 8082 udp, use something like this:

This will result in additional arguments to docker create/run to expose those internal ports, which will be mapped to random external ports. Any ports exposed in these docker options are in addition to ports exposed through EXPOSE commands in the docker image, and the ports 22 or 8080 which may be opened automatically for SSH or Jupyter.

After the instance has loaded, you can find the corresponding external public IP by opening the IP Port Info pop-up (button on top of the instance) and then looking for the external port which maps to your internal port. It will have a format of PUBLIC_IP -> INTERNAL_PORT. For example:

In this case, the public IP 65.130.162.74:33526 can be used to access anything you run on port 8081 inside the instance. We strongly recommend you test your port mapping.

You can quickly test your port mapping with a simple command to start a minimal web server inside the instance with the following command:

Which you would then access in this example by loading 65.130.162.74:33526 in your web browser. This should open a file directory.

In some cases you may need an identity port map like 32001:32001 where external are the same.

For this just use an out-of-range port above 70000:

These out of range requests will map to random external ports and matching internal ports. You can then find the resulting mapped port with the appropriate env variables like : $VAST_TCP_PORT_70000

A special instance api key should already be installed in your container and mapped to the env variable CONTAINER_API_KEY.

The vastai CLI may already be installed, but if not you can easily install with pip:

Then test it by starting the instance (which is a no-op as the instance is already running):

The instance apikey should already set, but if not, may need to specify it from env variable:

If that works then you can stop the instance as well:

You can also use destroy instance and a few other commands using the instance API key.

If $CONTAINER_ID and or $CONTAINER_API_KEY is not defined check your environment variables using the 'env' command. If you are missing the predefined env variables from an ssh session you may need to add a command to export them to /etc/environment (see earlier section on env variables).

If you don't have the instance api key for whatever reason, you can also generate it. First run the following from inside the instance to create a special per instance api key and save it in the appropriate location:

Then you should be able to run start/stop without passing in the key: