Performance Testing

endpoints

A lower level organization that lives within an Endpoint. It consists of a template (with extra filters for search), a set of GPU instances (workers) created from that template, and hyperparameters.

Worker_Groups

The highest level clustering of instances for the Serverless engine, consisting of a named endpoint string, a collection of Workergroups, and hyperparameters.

Endpoints

The Vast PyWorker is a Python web server designed to run alongside a machine learning model instance, providing Vast.ai Serverless compatibility.

PyWorker

Workergroup

Endpoint

The minimum number of workers you want to keep "cold" (meaning stopped and fully loaded) when your group has no load.

min_cold_workers

The maximum number of workers your router group can have.

max_workers

Route

Serverless

Overview

Getting Started With Serverless

Architecture

Pricing

Serverless Parameters

Debugging

Inside a Serverless GPU

Creating New PyWorkers

pyworker

Logs

Worker List

Create Endpoints and Workergroups

vLLM

Text Generation Inference (TGI)

Comfy UI

Untitled

Python webserver that collects metrics for the autoscaler.

Pyworker