Worker List

Untitled

A lower level organization that lives within an Endpoint. It consists of a template (with extra filters for search), a set of GPU instances (workers) created from that template, and hyperparameters.

Worker_Groups

The highest level clustering of instances for the autoscaler, consisting of a named endpoint string, a collection of Worker groups, and hyperparameters.

Endpoints

The Vast PyWorker is a Python web server designed to run alongside a machine learning model instance, providing autoscaler compatibility.

PyWorker

Worker_Group

Endpoint

The minimum number of workers you want to keep "cold" (meaning stopped and fully loaded) when your group has no load.

min_cold_workers

The maximum number of workers your router group can have.

max_workers

Text Generation Inference (TGI)

Serverless

Overview

Architecture

Getting Started

Debugging

Inside an Autoscaler GPU

Creating New PyWorkers

Performance Testing

pyworker

Route

Logs

endpoints

autoscaler

Comfy UI

Python webserver that collects metrics for the autoscaler.

Pyworker