Guide · Intermediate · AI Tools
Local AI on Raspberry Pi for Offline Maker Tools (Practical Limits)
Offline assist is for privacy and lab continuity — not for matching cloud frontier models on a $35 board. Stack map: see also the directory pages for local/offline AI and Ollama. What “works” vs what “wow” is …
Project template
- Goal & materials
- Steps / firmware
- Troubleshooting
- AI assist notes
- Related gear & books
Stack map: see also the directory pages for local/offline AI and Ollama.
What “works” vs what “wow” is
| Goal | Pi-class reality | Better hardware |
|---|---|---|
| Chat for code comments / short prompts | 7B Q4-class models, patience required | x86 + GPU or M-series laptop |
| Whisper captions for short clips | tiny/base models OK | Desktop GPU for long-form |
| Image generation | Generally a bad time on Pi | ComfyUI on a real GPU box |
| Always-on lab helper | Excellent if expectations are humble | — |
Recommended path (Pi 5 class)
- 64-bit Pi OS, cooling that actually moves air, quality PSU.
- Install Ollama (or follow current ARM instructions) and pull a small model first (
llama3.2:1b/phi3:miniclass — names change; pick small). - Prove CLI chat before adding UIs.
- Add Open WebUI only if multiple humans need a browser front-end.
- Point Continue.dev at the local OpenAI-compatible endpoint for offline IDE assist.
Resource budget (order-of-magnitude)
| Resource | Guidance |
|---|---|
| RAM | 8 GB Pi minimum for less pain; leave headroom for the OS |
| Storage | Avoid tiny SD cards; models are large — USB SSD / NVMe preferred |
| Thermals | Throttle kills “interactive” feel; log vcgencmd measure_temp under load |
| Power | Official PSU; flaky power looks like software bugs |
Maker use-cases that make sense offline
- Rewrite error logs into checklists (still verify against docs).
- Draft OpenSCAD parameter blocks with strict prompts.
- Summarize your own NOTES.md from a failed print night.
- Caption a 3-minute bench video with Whisper tiny/base.
Anti-goals
- Expecting ChatGPT-4-class coding agents at 30 tok/s on a Pi 4.
- Exposing Open WebUI to the internet “for convenience.”
- Storing API keys for cloud models on a Pi you treat as disposable.
Export it like code
pi-local-ai/
MODELS.md # exact tags pulled + quant notes
docker-compose.yml # if you containerize Open WebUI
prompts/ # your house style prompts
benchmarks.md # tokens/sec + temp under load
Verdict
Yes — run local AI on a Pi for private, always-on, small-model assist in a lab. No — do not market it as a cloud replacement. When you need speed, move the model to a workstation and keep the Pi as the MQTT/brain for the house.