Local & Private AI

Cloud AI is fast and powerful. But it comes with trade-offs that matter for many users and organizations. Here are the three core reasons to choose local:

When you download a local AI model, you're downloading a GGUF file — a binary format developed by the llama.cpp project. GGUF stands for GPT-Generated Unified Format. It packs everything the inference engine needs into a single file:

Ollama and LM Studio both use GGUF under the hood via the llama.cpp inference engine, which has been optimized to run on CPUs (including Apple Silicon's Metal and standard x86 AVX2) without requiring a discrete GPU. You can find GGUF models on Hugging Face by searching for “GGUF” + any model name.

A full-precision AI model stores each weight as a 16-bit or 32-bit floating point number. Quantization compresses these to fewer bits — trading a small amount of accuracy for a dramatic reduction in file size and RAM usage. The format code tells you exactly what kind of compression was used.

Ollama's architecture loads as many model layers as possible into GPU VRAM, then falls back to CPU for the rest. This means you can run models even without a discrete GPU — it's just slower. The limiting factor is total memory (RAM + VRAM combined).

GPU acceleration: Ollama automatically detects and uses NVIDIA CUDA (driver 550+; compute capability 5.0–6.2 needs driver 570+), AMD ROCm, and Apple Metal. On Apple Silicon, the unified memory architecture means there's no separate VRAM — all RAM is available for the model, which gives MacBooks a significant advantage over similarly specced Windows laptops.

Ollama is an open-source tool that wraps llama.cpp in a clean CLI and local REST API (port 11434). It handles model download, storage, hardware detection, and inference — all automatically.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download the installer from ollama.com

ollama run mistral
# Downloads on first run, then opens a chat prompt

ollama list              # Show downloaded models
ollama pull gemma       # Download without running
ollama rm mistral       # Delete a model
ollama ps               # Show currently running models
ollama show gemma       # Show model info and license

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ollama" \
  -d '{
  "model": "mistral",
  "messages": [{"role": "user", "content": "Explain GDPR in 3 sentences."}],
  "stream": false
}'

LM Studio is a desktop app (Windows, macOS, Linux) that wraps the same GGUF/llama.cpp stack in a graphical interface. No terminal required. It's the better starting point if command-line tools feel unfamiliar.

The local LLM landscape in 2026 has matured significantly. Here are the top options by use case and hardware tier, based on MMLU, HumanEval, and IFEval benchmarks:

# 4–8 GB RAM
ollama pull phi4-mini

# 8–16 GB RAM
ollama pull gemma3:12b

# 16–32 GB RAM (128K context; default mistral-small tag is 24b/32K)
ollama pull mistral-small:22b

# 40+ GB RAM
ollama pull llama3.3:70b
ollama pull qwen2.5:72b

Running AI locally is not a magic privacy guarantee — but it is a meaningful one. Here's what you actually get:

The right mental model: Local AI is not “worse cloud AI.” It's a different tool with different trade-offs. Many professionals use both: local models for privacy-sensitive tasks and rough drafts, cloud models for final output quality where data is non-sensitive.

Local models respond well to explicit structure. Use these templates as starting points — copy directly into Ollama or LM Studio:

You are a professional analyst. Summarize the following document for an internal audience.

Requirements:
- 3–5 bullet points maximum
- Flag any risks or open questions
- Do not add information not present in the document
- Use plain language

Document:
[PASTE DOCUMENT HERE]

You are a senior software engineer conducting a code review. Review the following code for:
1. Security vulnerabilities (SQL injection, XSS, SSRF, etc.)
2. Logic errors or edge cases not handled
3. Performance issues
4. Adherence to clean code principles

For each issue found: explain what it is, why it matters, and how to fix it.

Code to review:
[PASTE CODE HERE]

You are an executive assistant. Convert these raw meeting notes into a structured document.

Output format:
## Summary (2-3 sentences)
## Key Decisions
## Action Items (owner, due date)
## Open Questions
## Next Meeting

Raw notes:
[PASTE NOTES HERE]