Creating a Custom Template

how to create your own template introduction this guide will help you create custom templates for running containers on vast ai there are three approaches using provisioning script with recommended templates (simplest) create dockerfile from vast ai base images (recommended for custom images) using existing docker images (with security considerations) vast instances are currently linux docker containers and ubuntu virtual machines many linux based images should work within vast prerequisites a vast ai account ssh client installed on your local machine and ssh public key added in the ssh keys tab in the keys section of your console (optional) install and use vast cli approach 1 using provisioning script (simplest) this approach lets you customize recommended templates without building a custom image steps go to the templates tab in vast ai interface search for "base image" or "pytorch" depending on your needs vast ai/base image is a general purpose image vast ai/pytorch is a base image for working with pytorch based applications on vast click "edit" on your chosen template add the provisioning script environment variable in the environment variables section, add a new variable named "provisioning script" the value should be a url pointing to a shell script (from github, gist, etc ) if you're specifying a shell script stored in github repo, specify url like url https //raw\ githubusercontent com/karthik vast ai/vast cli/distributed inference integration/provisioning script sh7 make sure to click "+" to add the environment variable click create and use example provisioning script \#!/bin/bash \# cd /workspace/ \# cause the script to exit on failure set eo pipefail \# activate the main virtual environment /venv/main/bin/activate \# install your packages pip install your packages \# download some useful files wget p "${workspace}/" https //example org/my application tar gz tar xvf ${workspace}/my application tar gz" \# set up any additional services echo "my supervisor config" > /etc/supervisor/conf d/my application conf echo "my supervisor wrapper" > /opt/supervisor scripts/my application sh chmod +x /opt/supervisor scripts/my application sh \# reconfigure the instance portal rm f /etc/portal yaml export portal config="localhost 1111 11111 /\ instance portal|localhost 1234 11234 /\ my application" \# reload supervisor supervisorctl reload this script will run on first boot to set up your environment all installations should go to /workspace/ for proper persistence configuring application access with portal config the base image template includes portal config for secure application access management this environment variable controls how applications are exposed and accessed portal config structure hostname\ external port\ local port\ url path\ application name|hostname\ external port\ local port\ url path\ application name the structure of this variable is each application is separated by the | character each application parameter is separated by the character each application must specify hostname\ external port\ local port\ url path\ application name example "localhost 8002 18002 /hello\ myapp|localhost 1111 11111 /\ instance portal|localhost 8080 18080 /\ jupyter|localhost 8080 8080 /terminals/1\ jupyter terminal|localhost 8384 18384 /\ syncthing|localhost 6006 16006 /\ tensorboard" the hostname in docker instances will always be localhost where the internal port and local port are not equal then caddy will be configured to listen on 0 0 0 0\ external port acting as a reverse proxy for hostname\ local port if the external port and local port are equal then caddy will not act as a proxy but the instance portal ui will still create links this is useful because it allows us to create links to jupyter which is not controlled by supervisor in jupyter launch mode url path will be appended to the instance address and is generally set to / but can be used to create application deep links the caddy manager script will write an equivalent config file at /etc/portal yaml on boot if it does not already exist this file can be edited in a running instance important when defining multiple links to a single application, only the first should have non equal ports we cannot proxy one application multiple times note instance portal ui is not required and its own config declaration can be removed from portal config this will not affect the authentication system approach 2 create dockerfile from vast ai base images why use vast ai base images? built in security features through caddy proxy automatic tls encryption for web services authentication token protection proper isolation between external and internal services steps start with a vast ai base image or vast ai pytorch base image in your dockerfile \#for example from vastai/base image\ cuda 12 6 3 cudnn devel ubuntu22 04 py313 \# or from vastai/pytorch 2 6 0 cuda 12 6 3 py312 \# install your applications into /opt/workspace internal/ \# this ensures files can be properly synced between instances workdir /opt/workspace internal/ \# activate virtual environment from base image run /venv/main/bin/activate run your installation commands build and push your image to a container registry create a new template using your custom image see next approach for steps on how to do that approach 3 create template using existing docker images you can use existing docker hub images, but please note running external facing services without proper security measures is not recommended consider wrapping the service in vast ai's security infrastructure step 1 navigate to the 'templates' section once logged in, click on templates to go to the template section this is where you can create, manage, and edit templates step 2 create a new template click the button + new to create a new template create step 3 specify the image path/tag of your docker image in the image path tag, you can add the path/tag of the docker image you wish to use the version tag is where you would set the version image\&tag step 4 set environment variables you can use the e option to set environment variables in the docker options field by adding something like e myvar1 you can also add environment variables one by one in the environment variables section these entries will automatically sync with the docker options field make sure to click "+" adding the environment variable and value env\&stuff these options will be appended to the docker run command before the docker run command is executed within one of the machines similar to how it would be if you were executing "docker run e myvar1 image\[ |@digest]" on your local machine docker run env option documentation https //docs docker com/reference/cli/docker/container/run/#env step 5 \[optional] open custom ports in addition to setting environment variables, you can also open custom ports by adding p ${host machine port} ${container port} to the docker options section you can also add them to the ports section once more, this will automatically sync with the docker options field env\&stuff faq docid fgh0zckklw1nz3kttmal you can see what's happening inside a container at particular port by finding the public ip address mapped to that container port instances guide docid\ v5zbddmwmtwvghat6wnu instances guide docid\ v5zbddmwmtwvghat6wnu internally, one of the machine's open ports is mapped to this container port you can use ssh to open a ssh connection to the host machine's public ip address the machine will forward traffic from the host machine's public port to the container port you specified you can use a ssh command like the one below ssh p \[ssh port] \[user]@\[remote host] l \[local port]\ localhost \[remote port] ssh p 22 user\@remote example com l 8080\ localhost 8080 step 6 pick a launch mode you can choose the jupyter, ssh, or entrypoint launch mode depending on your needs vast launch modes https //docs vast ai/instances/launch modes step 7 add starting bash commands to on start script section these commands will be executed as soon the docker container starts, similar to how a bash script be would be executed suppose you created a bash script and added a cmd command in the dockerfile so docker would execute this bash script when the container starts before vast \# use a base image from ubuntu 20 04 \# make the script executable run chmod +x /usr/local/bin/start sh \# set the script as the default command cmd \["/usr/local/bin/start sh"] in this case cmd specifies the default command to run when the container starts, which is your start sh script in on start script section with vast chmod +x /usr/local/bin/start sh bash /usr/local/bin/start sh you can also try to overwrite existing files built into the image make sure you can switch to a user that has write permissions to that particular file for example, you can remove all instances of ' sslonly' in a particular file using sed sed i 's/ sslonly//g' /dockerstartup/vnc startup sh you can also make directories sudo i u kasm user mkdir p /home/kasm user/desktop exampleonstart make sure to append environment variables to /etc/environment file in your on start section because this makes environment variables available to all users and processes and ensures they are persisted even if your instance/docker container is rebooted this code snippet will take care of this env >> /etc/environment also make sure to find the image’s entrypoint or cmd command and call that command at the end of the on start section we overwrite that command to set up jupyter/ssh server, etc under the hood step 8 specify docker repository server name if necessary you don't have to specify docker io as the server name if your repository is docker hub docker automatically uses docker io to pull the image if no other registry is specified you do have to specify your server name if your repository is something else for example, github container registry (ghcr) server name ghcr io google container registry (gcr) server name gcr io step 9 specify docker login username and password we will have the same character set for docker login usernames and passwords as docker soon dockerlogin step 10 specify template name and description templatename step 11 select the amount of disk space you want diskspace2 step 12 fill out accessibility section choose whether to include readme, make template public, and fill out readme if you desire readme step 13 choose to save and/or select your template a new template will be created when you edit, make changes, and save createbuttons \[optional] download/upload files/backup to cloud storage you can download and upload files copy data to/from instance storage https //docs vast ai/instances/data movement you can also backup data to cloud storage backup data to cloud storage https //docs vast ai/instances/cloud sync troubleshooting if your image is built for a different cpu architecture than your vast machine, then it won't work you can try to find a machine with the required cpu architecture using our gui or cli cli documentation commands docid\ lxgndpzugf1ini1sq2gt0 if your image requires a higher cuda version, then look for a machine with a higher max cuda version the max cuda version can be found on the instance card instances guide docid\ v5zbddmwmtwvghat6wnu if your image is built to run jupyter, then try running it on a port different than 8080 if you are having issues using ssh launch mode, add your public key to / authorized keys, install openssh, start openssh when the container starts, and forward the ssh server's port template examples creating templates for grobid