Endpoint Logs
Endpoint logs are available under the “All Workergroups” tab in the Serverless endpoint, within the Vast console UI.
Endpoint Log
- Worker creation, starts, stops, and destruction
- Worker state changes and the reason for each change, for example
Worker 67890: model_loading -> idle (received loadtime) - The error reason when a worker enters the Error state, and any stack trace reported by the PyWorker
- Notices when a worker is replaced because its machine’s rental contract is ending
- Summarized performance for all workers and workergroups
- Measured and estimated performance and worker load
- Marketplace offer details used in worker recruitment
Worker Logs
Worker logs are accessible on a per-worker basis. To view worker logs, navigate to the serverless endpoint in question and click on this icon next to the worker.
Worker Log
Log Characteristics and Retention
- Logs are streaming outputs, typically only a few seconds behind real-time.
- Logs are not permanently maintained and are intended for near real-time debugging of issues. Each endpoint and workergroup keeps up to about 16 MB of recent history per log level.
- Users who need longer retention should periodically download logs to store them externally.
Accessing Logs Through the CLI
In addition to accessing these logs through the UI, you can use the vastai CLI to check endpoint and worker group logs at different log levels:
Use
--tail to set how many bytes to return from the end of the log. The default is 10,000 bytes. The same logs are available from the get endpoint logs and get workergroup logs APIs.
Endpoint logs
CLI Command
Workergroup logs
CLI Command