Initial Rollout
The serverless engine learns the cost-vs-performance profile of each GPU class in yoursearch_params from real workers running real traffic (see Choosing GPUs). The speed at which it “settles” into the most cost-effective mix depends on how quickly workers are recruited and released, so it helps to apply a test load during the first day of operation to give the engine enough signal to converge.
Best practice is to scale to double the number of expected required workers, then back down, three separate times.
Simulating Load
For examples of how to simulate load against your endpoint, see the client examples in the Vast SDK repository: https://github.com/vast-ai/vast-sdk/blob/main/examples/client/vllm_load_example.pyManaging for Bursty Load
- Adjust
cold_workers: This will change the number of managed inactive workers, and increase capacity for high peak - Check
max_workers: Ensure this parameter is set high enough for the serverless engine to create the necessary number of workers
Managing for Low Demand or Idle Periods
- Adjust
min_load: Reducingmin_loadwill reduce the minimum number of active workers. Set to1to reduce the number to its minimum value of 1 worker, or set to0to put all workers into inactive states. - Adjust
cold_workers: This will change the number of managed inactive workers
Scaling to Zero
When traffic stops, the endpoint scales down on its own. How far it goes depends on the floors you set:- To scale to zero active workers while keeping inactive workers available for fast reactivation, set
min_load = 0. Active workers are stopped as short-term demand fades. To control exactly how long the endpoint can sit idle before that happens, set a positiveinactivity_timeout. - To scale to zero total workers, also set
cold_workers = 0andmin_cold_load = 0. The inactive pool is sized from the endpoint’s long-term average demand, so inactive workers are destroyed gradually over the following hours, not at the inactivity timeout. This minimizes cost during extended idle periods but incurs cold-start latency when traffic resumes. - To keep at least one worker active at all times, set
min_loadabove0.
Workers with a Contract End Date
Some machines on the marketplace can only be rented until a set end date. When a worker’s rental contract is about to end, the serverless engine replaces it automatically:- Shortly before the end date, the engine recruits a replacement worker. It starts the replacement early enough for it to load, based on how long workers in the workergroup usually take, and at most one hour ahead.
- While the replacement loads, it does not receive requests and does not count toward
max_workers. - Once the replacement is ready, it takes over. The old worker stops receiving new requests and is destroyed when its in-progress requests finish, after at most 5 minutes, or at the end date, whichever comes first.
Preparing replacement for worker 67890: contract expires in 1800s, expected load time 600s. A worker that reaches its end date without a replacement is destroyed, and the engine recruits capacity as usual. For billing during the overlap, see Pricing.
Managing Queue Time
Usemax_queue_time and target_queue_time to control how the autoscaler responds to request queuing:
- Increase
max_queue_timeto allow more requests to buffer on each worker before the system holds them in the global queue. This is useful for workloads with predictable, longer processing times. - Decrease
target_queue_timeto trigger more aggressive scale-up when queue times rise, reducing latency at the cost of potentially higher worker counts. - Increase
target_queue_timeto tolerate higher queue times before scaling up, reducing costs when some latency is acceptable.