Video Generation

video generation guide using comfyui on vast ai this guide will walk you through setting up and using comfyui for video generation on vast ai comfyui provides a powerful node based interface for creating advanced stable diffusion pipelines, making it ideal for video generation workflows prerequisites a vast ai account basic familiarity with image or video generation models (optional) read jupyter guide (optional) ssh client installed on your local machine and ssh public key added in account tab at cloud vast ai setting up your instance 1\ select the right template navigate to the templates tab to view available templates for video generation, we recommend searching for "comfyui" among the recommended templates the comfyui template provides a powerful and modular stable diffusion gui for designing and executing advanced pipelines using a graph/nodes/flowchart based interface template features access through both jupyter and ssh instance portal token based authentication enabled by default built in provisioning script for models and custom nodes 2\ edit your template configuration add/update these environment variables as needed \# core settings comfyui args=" disable auto launch port 18188 enable cors header" # comfyui launch arguments \# authentication web enable https=false # enable/disable direct https web enable auth=true # enable/disable authentication \# access tokens cf tunnel token="" # cloudflare zero trust token civitai token="" # access gated civitai models hf token="" # access gated huggingface models \# custom setup provisioning script="" # url to custom provisioning script provisioning script default script includes popular image models and custom nodes fully customizable create your own script for a custom instance must be bash compatible and start with #!/bin/bash upload modified script to a github gist or respository and update the provisioning script variable to point to the raw file ⚠ important never save your template as public if you've included tokens or other secrets in the docker options field select your template from ' my templates ' after making any desired edits to it 3\ create your instance in the search interface , look for machines that have sufficient vram to handle your chosen video model ⚠ all models are different so check the model requirements carefully click rent to create an instance on the machine with the gpu of your choice 3\ connect to your instance go to instances tab to see your instance loading when the blue button says "open", click this button to access the instance portal which will provide access to comfyui and other useful applications click the direct link or cloudflare quick tunnel link to access comfyui here's a beginner's guide to using comfyui 4\ select a video workflow comfyui has a workflow browser, so for a quick start you can choose on of their templates comfyui template workflows we'll pick the ltx video workflow for this guide simply click it to proceed 5\ download missing files your new instance will not yet have the required models, but fortunately comfyui will alert us to this and offer the models for download unfortunately, the interface does not know it is running in the cloud so clicking the download buttons will download the models to your local machine to work around this you can either download the models to your computer and then upload them to the instance ssh to the instance and use curl or wget to directly download the models to their correct locations to complete this guide we will use ssh \# download the models wget content disposition p /workspace/comfyui/models/checkpoints/ "https //huggingface co/lightricks/ltx video/resolve/main/ltx video 2b v0 9 safetensors" wget content disposition p /workspace/comfyui/models/clip/ "https //huggingface co/comfyanonymous/flux text encoders/resolve/main/t5xxl fp16 safetensors" \# download the workflow source image wget content disposition p /workspace/comfyui/input/ "https //comfyanonymous github io/comfyui examples/ltxv/island jpg" the above commands will download the required models into the instance when the downloads have completed you can refresh the browser window to clear the missing models error 6\ run the workflow finally, click the run button to process the workflow feel free to modify the prompts and experiement! pre configured templates we have some pre configured comfyui templates and one for this guide check them out here comfyui + ltx video open sora resources and further reading comfyui official repository vast ai documentation comfy workflows vast ai support chat on website remember to always check vram usage and adjust parameters accordingly start with smaller frames and resolutions, then scale up as you become more comfortable with the workflow