Worker Errors
The Vast PyWorker framework automatically detects some errors, while others may cause the instance to timeout. When an error is detected, the Serverless system will destroy or reboot the instance. To manually debug an issue, check the instance logs available via the logs button on the instance page in the GUI. All PyWorker issues will be logged here. If further investigation is needed, ssh into the instance and find the model backend logs location by running:Text
Text
Increasing Load
To handle high load on instances:- Set
test_workers
high: Create more instances initially for Worker Groups with anticipated high load. - Adjust
cold_workers
: Keep enough workers around to prevent them from being destroyed during low initial load. - Increase
cold_mult
: Quickly create instances by predicting higher future load based on current high load. Adjust back down once enough instances are created. - Check
max_workers
: Ensure this parameter is set high enough to create the necessary number of workers.
Decreasing Load
To manage decreasing load:- Reduce
cold_workers
: Stop instances quickly when the load decreases to avoid unnecessary costs. The serverless system will handle this automatically, but manual adjustment can help if needed.