Worker States in the Vast UI
The following table describes all available worker states that are shown to users in the UI:
For more information on the specific breakdown of charges, please refer to Serverless pricing
Worker States in the Vast CLI and logs
The worker logs, accessible through the Vast CLI and UI, have more detailed descriptions of worker states to aid in debugging issues. These are described in the table below. The value in parentheses is thestatus string written to the logs and returned by the worker status APIs (REST and WebSocket).
Each state change is written to the endpoint and workergroup logs as
Worker <id>: <old state> -> <new state> (<reason>).
Automatic Recovery
The serverless engine handles unhealthy workers without user action:- Error: Workers in the Error state are either restarted or destroyed and replaced. A Ready worker whose reliability (the share of routed requests it completes) falls below
0.5is moved to Error. - Timeouts: A worker that stays in a transitional state for too long is moved to Error. For example, a worker that is still Creating 5 minutes after its creation request succeeded, or still Stopping after 2 minutes.
- Unavailable: If the machine frees up, the worker returns to Inactive. When a start attempt fails because the GPUs are taken, the engine retries every 90 seconds. After 10 failed attempts the worker is destroyed, or sooner if the endpoint is at
max_workersand has no other inactive workers to start. While a worker is Unavailable, the engine may recruit other workers to cover its capacity. - Offline: If the machine reconnects, the worker resumes. A worker that stays Offline for 5 minutes is destroyed.