> ## Documentation Index > Fetch the complete documentation index at: https://docs.vast.ai/llms.txt > Use this file to discover all available pages before exploring further. # GLM-4.7-Flash # Deploying GLM-4.7-Flash on Vast.ai [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a 30B-parameter Mixture of Experts model from Zhipu AI that activates 3B parameters per token. Despite being roughly the same active parameter count as Qwen3-30B-A3B, GLM-4.7-Flash has a fundamentally different attention architecture: it uses 20 key-value heads with a 256-dimensional value head (compared to Qwen3's 4 KV heads with 128-dimensional values). This means its KV cache consumes approximately 10x more memory per token of context, making hardware selection critical for long-context deployments. This guide covers deploying GLM-4.7-Flash on Vast.ai using SGLang with 4x RTX 3090 GPUs. ## Why the High Memory Requirement GLM-4.7-Flash uses Multi-Head Attention (MHA) rather than the Grouped Query Attention (GQA) common in recent MoE models. This architectural choice affects KV cache size directly: | Parameter | GLM-4.7-Flash | Qwen3-30B-A3B | | -------------------------- | ------------- | ------------- | | num\_hidden\_layers | 47 | 48 | | num\_attention\_heads | 20 | 32 | | **num\_key\_value\_heads** | **20** | **4** | | hidden\_size | 2048 | 2048 | | **v\_head\_dim** | **256** | 128 | **KV cache per token:** * GLM-4.7-Flash: `2 x 47 x 20 x 256 x 2 bytes` = \~962 KB/token * Qwen3-30B-A3B: `2 x 48 x 4 x 128 x 2 bytes` = \~96 KB/token This means for 200k context, the KV cache alone requires \~188 GB. Combined with \~60 GB for model weights, full context deployment needs 250+ GB VRAM. ## What We're Deploying This guide uses the following configuration: * **GPUs**: 4x RTX 3090 (96 GB total VRAM) * **Context Length**: 8,192 tokens * **Disk**: 200 GB * **CUDA**: 12.2-12.6 * **Docker image**: `lmsysorg/sglang:dev-pr-17247`, must use this image; the `latest` tag lacks MLA support for GLM-4.7-Flash ## Prerequisites * A [Vast.ai](https://cloud.vast.ai) account with credits * Vast.ai CLI installed (`pip install vastai`) * Your Vast.ai API key configured (`vastai set api-key YOUR_API_KEY`) ## Step 1: Find an Instance Search for 4x RTX 3090 instances: ```bash theme={null} vastai search offers "gpu_name=RTX_3090 num_gpus=4 direct_port_count>=1 cuda_vers>=12.2 cuda_vers<=12.6" --order dph_base --limit 10 ``` **What these filters mean:** * `gpu_name=RTX_3090`: Target GPU type * `num_gpus=4`: Four GPUs for tensor parallelism * `direct_port_count>=1`: At least one direct port for API access * `cuda_vers>=12.2 cuda_vers<=12.6`: CUDA version range that avoids driver issues ## Step 2: Deploy the Model Generate an API key to secure your endpoint: ```bash theme={null} openssl rand -hex 32 ``` Save the output and set it as an environment variable: ```bash theme={null} GLM_API_KEY="" ``` Create an instance with SGLang serving GLM-4.7-Flash. Replace `` with the ID from Step 1: ```bash theme={null} vastai create instance \ --image lmsysorg/sglang:dev-pr-17247 \ --env "-p 8000:8000" \ --disk 200 \ --onstart-cmd "python3 -m sglang.launch_server \ --model-path zai-org/GLM-4.7-Flash \ --host 0.0.0.0 \ --port 8000 \ --tp-size 4 \ --context-length 8192 \ --trust-remote-code \ --dtype float16 \ --mem-fraction-static 0.85 \ --api-key $GLM_API_KEY" ``` **Key parameters:** * `--tp-size 4`: Distribute model across all 4 GPUs using tensor parallelism * `--context-length 8192`: Maximum sequence length (increase if you have more VRAM) * `--dtype float16`: Required for RTX 3090 which does not natively support bfloat16. Use `--dtype bfloat16` on A100/H100 * `--mem-fraction-static 0.85`: Allocate 85% of GPU memory for model and KV cache * `--trust-remote-code`: Required for the GLM-4.7-Flash architecture * `--api-key`: Secures the endpoint with bearer token authentication For longer context, use GPUs with more VRAM (like A100 or H100) and increase `--context-length`. A100/H100 also support `--dtype bfloat16`. ## Step 3: Monitor Startup The model is \~60 GB and takes 8-10 minutes to download on first deployment. Monitor progress with: ```bash theme={null} vastai logs ``` Look for these messages indicating progress: * `Loading model weights`, Download and loading in progress * `The server is fired up and ready to roll!`, Server is ready to accept requests Get your instance IP and port once it's running: ```bash theme={null} vastai show instance ``` ## Step 4: Send Requests ### Using curl ```bash theme={null} # Health check curl http://:/health # List models curl http://:/v1/models \ -H "Authorization: Bearer $GLM_API_KEY" # Chat completion curl http://:/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $GLM_API_KEY" \ -d '{ "model": "zai-org/GLM-4.7-Flash", "messages": [{"role": "user", "content": "Write a haiku about GPU computing"}], "max_tokens": 100 }' ``` **Example response:** ```json theme={null} { "choices": [{ "message": { "role": "assistant", "content": "Parallel threads,\nCalculated at once,\nRise from the shadows." }, "finish_reason": "stop" }], "usage": {"prompt_tokens": 12, "completion_tokens": 73} } ``` ### Using OpenAI SDK ```python theme={null} from openai import OpenAI client = OpenAI( base_url="http://:/v1", api_key="" ) response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[{"role": "user", "content": "Write a haiku about GPU computing"}], max_tokens=100 ) print(response.choices[0].message.content) ``` **Example output:** ``` Parallel threads, Calculated at once, Rise from the shadows. ``` ## Cleanup Destroy your instance when finished to stop charges: ```bash theme={null} vastai destroy instance ``` ## Conclusion GLM-4.7-Flash offers strong reasoning and coding capabilities in a 3B active parameter footprint. The trade-off is its attention architecture-using 20 KV heads instead of 4 means you need more VRAM per token of context than similarly-sized MoE models. For applications that need 8k context windows, 4x RTX 3090 provides a low-cost deployment option. For longer context requirements, scaling to A100 or H100 instances allows you to increase `--context-length` proportionally with available VRAM. ## Next Steps * **Increase context**: Use GPUs with more VRAM (like A100 or H100) to serve longer context windows * **Add load balancing**: Use [SGLang Router](/examples/serving-infrastructure/sglang-router-vast) to distribute requests across multiple instances ## Additional Resources * [GLM-4.7-Flash Model Card](https://huggingface.co/zai-org/GLM-4.7-Flash), Model weights and architecture details * [SGLang Documentation](https://sgl-project.github.io/), SGLang server configuration and features * [SGLang Docker Images](https://hub.docker.com/r/lmsysorg/sglang/tags), Available Docker tags including dev builds * [Vast.ai CLI Guide](/cli/hello-world), Complete CLI reference for managing instances