autoscaler

A lower level organization that lives within an Endpoint. It consists of a template (with extra filters for search), a set of GPU instances (workers) created from that template, and hyperparameters.

Worker_Groups

The highest level clustering of instances for the Serverless engine, consisting of a named endpoint string, a collection of Workergroups, and hyperparameters.

Endpoints

The Vast PyWorker is a Python web server designed to run alongside a machine learning model instance, providing Vast.ai Serverless compatibility.

PyWorker

Workergroup

Endpoint

The minimum number of workers you want to keep "cold" (meaning stopped and fully loaded) when your group has no load.

min_cold_workers

The maximum number of workers your router group can have.

max_workers

Overview

Serverless

Getting Started With Serverless

Architecture

Pricing

Serverless Parameters

Debugging

Inside a Serverless GPU

Creating New PyWorkers

Performance Testing

pyworker

Route

Logs

Worker List

Create Endpoints and Workergroups

endpoints

vLLM

Text Generation Inference (TGI)

Comfy UI

Untitled

Python webserver that collects metrics for the autoscaler.

Pyworker