What LLM Can I Run Locally? A Guide by RAM, VRAM, and Use Case
- What LLM Can Your Computer Run?
- Which Model Should You Choose for Your Task?
- How to Run Your Chosen Model Locally
- What If the Model Is Too Slow or Won’t Load?
- FAQ
You can run a local LLM if your computer has enough available memory for the model, its context, and the inference software. Start by checking your RAM, GPU VRAM, and operating system, then choose a supported quantized model. The right choice is not simply the largest model that loads. It is one that responds fast enough and handles your tasks reliably.
Last checked: November 2026. Model tags change fast, so confirm them in the Ollama library before you download.
What LLM Can Your Computer Run?
Check three things first:
- System RAM: Check your total installed memory (this matters most for CPU-only systems).
- Graphics card VRAM: Look up your exact GPU model and its dedicated VRAM.
- Apple Silicon Macs: Note your total unified memory.
Then find your tier below. Every pick comes with a specific tag you can copy straight into Ollama.
Hardware Tier Guide
| Hardware Tier (VRAM / Unified Memory) | Try This First (Ollama Tag) | Good For | What to Know |
| 8GB system RAM (no GPU / entry-level Mac) | Qwen3.5:2b (Q8_0, ~2.7GB) | Quick Q&A, rewording text | macOS and Windows need 3.5–4.5GB for system tasks. A 2B model is the safest choice to avoid out-of-memory (OOM) crashes and heavy swapping. Limit your context length. |
| 16GB system RAM (CPU-only inference) | Qwen3.5:4b (Q4_K_M, ~3.4GB) Larger option: Qwen3.5:9b (~6.6GB) | Everyday text tasks | Runs on the CPU only, so expect slower token generation. Limit the context (num_ctx) to 4K–8K so the growing KV cache doesn’t push you into swap. |
| 8–12GB VRAM (e.g., RTX 3060/4060) | 8GB: gemma4:e4b-it-qat or qwen3.5:8b (~6.1GB) 12GB: gemma4:12b-it-qat (~7.2GB) | Daily assistant, light coding | On 8GB cards, keep the context window moderate to prevent VRAM from spilling over into system RAM. 12GB cards handle 12B models comfortably. |
| 16GB VRAM (e.g., RTX 4080 16GB) | gpt-oss:20b (Q4_K_M, ~12.5GB) Alternative: gemma4:12b-it-qat | Demanding coding, deep Q&A | Fits 16GB cards with a standard context window. If you need long context, consider dropping down to a 12B model. |
| 24GB VRAM / 24GB Unified Mac (e.g., RTX 4090 / 24G Mac) | gemma4:26b-a4b-it-qat (~16GB) Coding: qwen3-coder:30b (~19GB) | Heavy multitasking, structured code | On a 24GB GPU, a 30B model leaves only ~2–3GB for the KV cache. If you need 16K+ context for long codebase analysis, choose a 12B–20B model instead. |
| 32GB+ Unified Memory Mac (or multi-GPU setups) | qwen3-coder:30b (Q4_K_M, ~19GB) gemma4:26b-a4b-it-qat | Large codebase analysis, long context | 32GB Macs and high-end setups run these comfortably and leave plenty of room for a large KV cache. |
Which Model Should You Choose for Your Task?
If several models fit your machine, choose based on your main workload, or browse more local AI models to compare options.
- General chat and writing: gemma4:12b-it-qat. It strikes an excellent balance between coherence and reasoning. On 8GB VRAM cards, drop down to gemma4:e4b-it-qat or qwen3.5:8b.
- Coding: qwen3-coder:30b (Q4_K_M, 19GB). It’s built specifically for software engineering and fits 24GB GPUs (with short- to medium-context) or 32GB Macs. If you only have 16GB of VRAM or need long context on a 24GB card, use gpt-oss:20b or gemma4:12b-it-qat instead.
- Document Q&A and analysis: qwen3.5:9b. It handles extended contexts efficiently. When processing large documents, explicitly set a large enough context size (num_ctx) in your runtime configuration, and make sure you have enough free RAM or VRAM to hold the KV cache.
- Multimodal / vision: gemma4:12b-it-qat. It supports both text and image input through Ollama. For cards with less VRAM, try a compact dedicated vision model such as qwen2.5-vl:3b.
How to Run Your Chosen Model Locally
Method 1: LM Studio (Easiest Graphical Interface)
- Download and install: Get LM Studio and open the app.
- Search for models: Search for the model family directly (e.g., “Qwen3.5 9B” or “Gemma 4 12B”). LM Studio automatically searches Hugging Face and GGUF repositories.
- Select a format: Download the GGUF in Q4_K_M (or the native QAT-derived GGUF variant, if available). You can find Qwen3.5 GGUF models in a range of sizes.
- Load and test: Load the model into memory and try a prompt (for example, summarizing a short document) to check speed and memory headroom.
Method 2: Ollama (Command Line)
1. Run the command directly in your terminal:
Bash
# Text & Vision
ollama run gemma4:12b-it-qat
# Coding
ollama run qwen3-coder:30b
# Multimodal input example (image + text prompt)
# Note: Use an absolute path, or pass the image in interactive mode
ollama run gemma4:12b-it-qat "Describe this image: /absolute/path/to/image.png"
2. To monitor memory and VRAM usage while a model is running, open a new terminal window and run:
Bash
ollama ps
Tips: Prefer to skip the command line? iBoomto AI gives you a chat interface and one-click local model installation, so you can get started quickly, for free.
What If the Model Is Too Slow or Won’t Load?
| Problem | Root Cause | What to Do Next |
| Won’t load / crashes | Out of VRAM or system RAM | Close background apps, check how much memory is available, or switch to a smaller model. |
| Out of memory on long prompts | KV cache exceeds the allocated buffer or VRAM | Shorten the context length (num_ctx), split large documents into smaller chunks, or move down to a smaller model tier. |
| Runs, but extremely slowly | Model is offloaded to CPU/RAM | Check that GPU acceleration is enabled in the settings. If you’re running on CPU only, switch to a smaller model (e.g., 2B–4B). |
| Output quality is weak | Model is too small, or the prompt is poor | Switch to a larger model or a domain-specific one (e.g., a dedicated coding model). Raising the quantization level (e.g., Q4 to Q8) brings only minor gains compared with using a larger model. |
FAQ
QAT (Quantization-Aware Training): A method in which the model is trained with quantization loss taken into account. In tools like Ollama and LM Studio, QAT models are still packaged in standard formats (such as GGUF) for local us
Was this helpful?
Thank you for your feedback!