How to Run an LLM Locally in 10 Minutes (llama.cpp Guide)

The fastest way to run an LLM locally in 2026: install llama.cpp with one package-manager command, start llama-server with one more, and you have a private, OpenAI-compatible AI server on localhost — no account, no API key, no data leaving your machine. This guide walks through the whole thing in about ten minutes, then shows the payoff we think is the best everyday use for a local model: powering AI dictation cleanup in Vowen so your entire voice-to-polished-text pipeline runs on your own computer.

Why run an LLM locally at all?

• Privacy. Prompts and text never leave your machine. For dictated emails, client notes, or anything confidential, that is the whole argument.
• It's free. No per-token billing, no monthly plan. Once the model is downloaded, you can run a million requests and it costs you nothing.
• It works offline. Planes, trains, flaky hotel Wi-Fi — a local model doesn't care. Pair it with offline speech to text and nothing in your workflow needs a connection.
• No rate limits or outages. Your server, your rules.

The trade-off used to be that local setup was painful and the models were weak. Neither is true anymore: installation is a one-liner, and small open models are now excellent at everyday text tasks — exactly the kind of work AI dictation cleanup needs.

What you need

Almost any recent computer works. For the small models this guide uses (3-4B parameters, ~2 GB download), 8 GB of RAM is enough and 16 GB is comfortable. Apple Silicon Macs are outstanding at this — llama.cpp uses the GPU via Metal automatically. On Windows, a modern CPU alone is fine for cleanup-sized tasks; an NVIDIA or AMD GPU makes it faster but is not required.

Step 1: Install llama.cpp

llama.cpp is the open-source engine behind most local AI tools — a single small program with no background services and no bundled app. Install it with your package manager:

macOS (Homebrew):

brew install llama.cpp

Windows (winget, in PowerShell or Terminal):

winget install llama.cpp

That's it. Both commands install llama-server, the piece we need. (Prebuilt binaries, including CUDA and Vulkan GPU builds, are also on the llama.cpp GitHub releases page if you prefer.)

Step 2: Start the server — one command

This single command downloads a model from Hugging Face on first run (cached afterwards) and starts serving it:

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

A few seconds of loading later, you have an AI server running at http://localhost:8080. What the pieces mean:

• -hf pulls a model straight from Hugging Face by repo name — no separate download step, no file paths.
• Llama 3.2 3B Instruct is a small, fast, instruction-following model — a sweet spot for text cleanup. The download is about 2 GB.
• Q4_K_M is the quantization — a compression level that keeps quality high while fitting comfortably in 8 GB of RAM.

Useful optional flags: --port 8081 if 8080 is taken, -c 4096 for a larger context window, and -ngl 99 to offload all layers to a GPU on Windows (Macs use Metal automatically).

Step 3: Test it

Open http://localhost:8080 in your browser — llama-server ships with a built-in chat UI, so you can talk to your model immediately. Prefer the terminal? The server speaks the OpenAI API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Say hello in five words."}]}'

Note what you did not need: an API key. It's your server on your machine. (You can require one with --api-key if you ever expose it beyond localhost.)

Picking a model

For dictation cleanup and everyday text tasks, small instruct models are all you need:

• Llama 3.2 3B Instruct — the default above. Fast, light, great at following "clean this text up" style instructions.
• Qwen or Gemma small instruct models (3-4B) — equally strong alternatives; search Hugging Face for their official GGUF repos and swap the -hf argument.
• 7-8B models — if you have 16 GB+ of RAM, these give noticeably better results on complex rewriting while still running well on consumer hardware.

The payoff: fully local AI dictation

Here's why we walked you through this. Vowen already transcribes your speech on-device — your audio never leaves your computer. Its optional AI enhancement step (fixing punctuation, stripping filler words, tidying sentences) normally uses a cloud provider with your own API key. Point it at your new local server instead, and the entire pipeline — voice, transcript, cleanup — runs on your machine:

1. In Vowen, open Settings and go to the AI provider setup.
2. Choose Custom API (it's labeled OpenAI-compatible — which is exactly what llama-server is).
3. Set Base URL to http://localhost:8080/v1.
4. Type any name into the model field (for example llama-3.2-3b) — llama-server answers with whatever model it has loaded.
5. Leave the API key blank — it's optional for the Custom API provider, and your local server doesn't need one.
6. Test the connection, save, and turn on Enhance transcription with AI — globally, or per hotkey, so one shortcut dictates raw and another dictates polished.

Now hold your hotkey, speak naturally — filler words, false starts and all — and watch clean, punctuated text land wherever your cursor is. No audio uploaded, no text sent to a cloud API, no subscription paying for either. This works the same on Mac and Windows, and the core dictation is free — see pricing for what's in Pro.

Alternatives: Ollama and LM Studio

llama.cpp is our pick because it's a single binary with nothing running in the background you didn't start. But two other routes are popular, and both also work with Vowen's Custom API provider:

• Ollama — a model manager built on the same engine. If you already use it, run a model and set Vowen's Base URL to http://localhost:11434/v1 with the model name you pulled (for example llama3.2).
• LM Studio — a full GUI app for browsing and running models, with a local server mode you can point Vowen at the same way.

Troubleshooting

• Responses are slow: use a smaller model or lower quantization (3B at Q4 is very fast on almost anything), and on Windows add -ngl 99 if you have a GPU.
• Port already in use: start with --port 8081 and update the Base URL to match.
• Connection test fails: make sure llama-server is still running in its terminal window, and that the Base URL includes /v1 and uses http:// (not https) for localhost.
• Out of memory: pick a smaller model or a lower quant (e.g. Q4_K_M instead of Q8_0), and close other heavy apps.

Ten minutes, two commands, and you own your AI stack — a private model server that costs nothing to run, works offline, and turns spoken thoughts into clean text without a single byte leaving your computer.