How to Run a Private, Local AI Assistant (Models on Your Own PC)

Most AI assistants ship your prompts — including your code, your secrets, and your conversations — to a cloud you don't control. For a lot of work, that's a non-starter. The good news: you can run a genuinely capable private, local AI assistant whose model runs on your own machine. Here's how, and when to safely allow the cloud.

Why run AI locally at all?

• Confidentiality — regulated data, client code, and credentials never transit a third party.
• No provider telemetry — with local inference, your prompts aren't logged, profiled, or used to train someone else's model.
• Offline capable — it works on a plane, in an air-gapped lab, or when the Wi-Fi dies.
• Cost control — local inference has no per-token bill.

This is not a niche worry. In Cisco's 2024 Data Privacy Benchmark Study of 2,600 privacy and security professionals across 12 countries, 48% admitted entering non-public company information into generative AI tools, and 27% said their organisation had banned generative AI applications altogether for the time being. Developers feel it too: in Stack Overflow's 2025 Developer Survey, roughly 81% agreed that the security and privacy of their data is a concern with AI agents. Running the model locally removes the third party from that equation entirely.

How a local AI assistant works

The model runs on your own hardware via a local runtime like Ollama. Instead of calling a cloud API, the assistant routes requests to ollama/llama3 (or any model you've pulled) on localhost. A capable laptop GPU handles 7B–8B models comfortably; bigger models want more VRAM but run fine on a desktop card.

Ollama is the de facto standard runtime here: it is MIT-licensed, its GitHub repository had about 183,000 stars at the time of writing, and Ollama's Windows documentation lists Windows 10 22H2 or newer as the requirement, serving its API on http://localhost:11434. Most local assistants, Jarvis included, talk to that endpoint rather than shipping their own inference engine.

Local AI tools compared

A runtime alone gives you a private model, not an assistant. The tools below are all good at keeping inference on your machine; they differ in what they do with it. Every one of them is worth trying, and several work together — Jarvis uses Ollama for its local models.

If you want private chat with a local model, Jan, LM Studio and GPT4All are excellent and free. If you want the assistant to do the work rather than describe it — run the command, edit the file, restart the container — while keeping the same local-first privacy, that is the gap Jarvis AI Assistant fills.

Local-only vs. hybrid: pick your privacy mode

Fully local is the strongest guarantee, but cloud models are still smarter for the hardest tasks. The right design lets you choose. Jarvis exposes three explicit modes, which you pick in the app's Settings:

STRICT — local AI models only

Every model call goes to local Ollama models, so your prompts are never sent to a cloud AI provider. Online tools such as web search still work when you use them. Suited to sensitive codebases. Choose STRICT in the app's Settings and point the default model at Ollama.

GUARDED — cloud with auto-redaction (recommended)

Cloud models are allowed, but a privacy layer strips credential-like strings such as API keys, passwords and tokens before a request is sent. Email addresses and phone numbers are not redacted. You get frontier-model quality without sending your secrets.

RELAXED — the default, no redaction

Prompts go to the cloud model you chose as they are. It suits non-sensitive work; switch to GUARDED or STRICT when your work needs it.

The honest trade-offs

Local models are smaller and slower than frontier cloud models, so complex reasoning may need more prompting or a bigger local model. Hardware matters — more VRAM means larger, sharper models. The practical sweet spot for most people is GUARDED for everyday work and STRICT for anything sensitive, switching with a single setting.

Getting started with Jarvis

Install Jarvis, install Ollama and pull a model (ollama pull llama3), then set your privacy mode in Settings. Choose STRICT to use only local models, or GUARDED when you want a cloud model with credentials stripped first. It installs from the Microsoft Store, uses your Jarvis account for licence access, and keeps your conversations and memory on your PC.