Local is a place, not a virtue
Running a model on your own machine
A local model is a file of weights on a computer you control, plus a program that runs them. The prompt stays there unless some other app sends it out. The electricity still comes out of the wall.
What it is
The weights are the model. The program that runs them — llama.cpp, Ollama, and others — is the engine. They are not the same object. A chat window is a third thing: the box you type into.
The limit is memory. The weights have to sit in RAM or, much faster, in the graphics card's own memory. A card that cannot hold the file cannot run that model at a useful speed, no matter how clever the prompt is.
Quantization stores those weights with less precision so a bigger model fits. The common packaged form is a GGUF file. You trade some accuracy for the fit. It is compression, not a free larger brain.
How it works
You pick a model that fits the machine, load it, and ask. The reply is computed on that machine. Nothing about the math requires a company account. An account appears only if the app you used decides to phone home.
A small model on a small machine can be the right tool for a narrow job: sorting, a first draft, a private note. It will not match a frontier model on hard reasoning. 'It runs here' is not 'it knows more.'
Updates are yours to do. A cloud model can change overnight without you touching it. A local file changes when you replace the file. That is control, and it is also a chore.
Why it matters
Privacy is the plain reason. A question that never leaves the house cannot be logged by the service you did not call. Read the app, not the slogan. Some 'local' apps still send the text out for the reply.
Local is not free and not always greener. The chip in the room draws power, and a large model on a small card can take longer and waste more than a short call to a machine built for it. Count the whole path.
Sources
- llama.cpp is the public project that made GGUF and CPU/GPU local inference ordinary. The format notes live with that project, not in a marketing page.
- A graphics card's memory size is a spec sheet number. If the model file does not fit, the rest of the claim does not matter.
See it in the City
- Edge Nodes
The City's version of this page: a mind that lives in the district and answers from what is already there. The fiction adds years of local context. The fact is the same shape — the work sits near the question.