Skip to main content

Initial Rollout

The serverless engine learns the cost-vs-performance profile of each GPU class in your search_params from real workers running real traffic (see Choosing GPUs). The speed at which it “settles” into the most cost-effective mix depends on how quickly workers are recruited and released, so it helps to apply a test load during the first day of operation to give the engine enough signal to converge. Best practice is to scale to double the number of expected required workers, then back down, three separate times.

Simulating Load

For examples of how to simulate load against your endpoint, see the client examples in the Vast SDK repository: https://github.com/vast-ai/vast-sdk/blob/main/examples/client/vllm_load_example.py

Managing for Bursty Load

  • Adjust cold_workers: This will change the number of managed inactive workers, and increase capacity for high peak
  • Check max_workers: Ensure this parameter is set high enough for the serverless engine to create the necessary number of workers

Managing for Low Demand or Idle Periods

  • Adjust min_load: Reducing min_load will reduce the minimum number of active workers. Set to 1 to reduce the number to its minimum value of 1 worker, or set to 0 to put all workers into inactive states.
  • Adjust cold_workers: This will change the number of managed inactive workers

Scaling to Zero

When traffic stops, the endpoint scales down on its own. How far it goes depends on the floors you set:
  • To scale to zero active workers while keeping inactive workers available for fast reactivation, set min_load = 0. Active workers are stopped as short-term demand fades. To control exactly how long the endpoint can sit idle before that happens, set a positive inactivity_timeout.
  • To scale to zero total workers, also set cold_workers = 0 and min_cold_load = 0. The inactive pool is sized from the endpoint’s long-term average demand, so inactive workers are destroyed gradually over the following hours, not at the inactivity timeout. This minimizes cost during extended idle periods but incurs cold-start latency when traffic resumes.
  • To keep at least one worker active at all times, set min_load above 0.
When a request arrives while no workers are active, the engine starts an inactive worker, or recruits a new one if the endpoint has no workers.

Workers with a Contract End Date

Some machines on the marketplace can only be rented until a set end date. When a worker’s rental contract is about to end, the serverless engine replaces it automatically:
  1. Shortly before the end date, the engine recruits a replacement worker. It starts the replacement early enough for it to load, based on how long workers in the workergroup usually take, and at most one hour ahead.
  2. While the replacement loads, it does not receive requests and does not count toward max_workers.
  3. Once the replacement is ready, it takes over. The old worker stops receiving new requests and is destroyed when its in-progress requests finish, after at most 5 minutes, or at the end date, whichever comes first.
The endpoint logs record each step, for example Preparing replacement for worker 67890: contract expires in 1800s, expected load time 600s. A worker that reaches its end date without a replacement is destroyed, and the engine recruits capacity as usual. For billing during the overlap, see Pricing.

Managing Queue Time

Use max_queue_time and target_queue_time to control how the autoscaler responds to request queuing:
  • Increase max_queue_time to allow more requests to buffer on each worker before the system holds them in the global queue. This is useful for workloads with predictable, longer processing times.
  • Decrease target_queue_time to trigger more aggressive scale-up when queue times rise, reducing latency at the cost of potentially higher worker counts.
  • Increase target_queue_time to tolerate higher queue times before scaling up, reducing costs when some latency is acceptable.