vllm

# vLLM Serving vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads. ## Before you start If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear. Ask the user which model to serve; if they have no preference, recommend the small default [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). Also ask what context length the workload needs. vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first: ```bash nvidia-smi # NVIDIA: GPU model and free VRAM (AMD ROCm: rocm-smi) python3 --version # a recent Python is required ``` The model must fit the available VRAM — model size and context length drive the serve flags below. ## Suggested workflow 1. Ask the user which model to serve; with no preference, recommend [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). 2. Pick the engine the user prefers: vLLM for high-throughput GPU serving; Ollama is the simple default and the choice on macOS or CPU-only machines. 3. Serve on a free port, with the tool-calling flags whenever agents will call it (see below). 4. Verify with `curl http://local...