Ollama turns your own laptop or build box into an inference server, which means auth logs, scan output and client configs can hit a model without ever leaving your network. Here is the setup, six workflows worth stealing, and the gotchas that quietly ruin the results.
Most AI tutorials assume you are allowed to paste your data into somebody else’s API. Plenty of us are not. Tap through the workflow below, then read the walkthrough:
// install, pull, confirm the GPU actually took it
# macOS and Windows: grab the app from ollama.com/download # Linux: run the official install script from ollama.com ollama run gemma4 # pulls on first run, opens a chat ollama ls # what is on disk ollama ps # what is loaded, and on GPU or CPU
// the log never leaves the host
tail -n 400 /var/log/auth.log | ollama run gemma4 \ "Group these failed auth attempts by source IP. Rank by burst rate, not raw count. Flag anything that looks like credential stuffing. Output a markdown table. Do not speculate about intent."
// stop parsing prose, enforce a schema
from ollama import chat
from pydantic import BaseModel
class Finding(BaseModel):
severity: str
asset: str
action: str
res = chat(
model='gpt-oss',
messages=[{'role': 'user', 'content': 'Summarize this finding'}],
format=Finding.model_json_schema(),
options={'temperature': 0},
)
finding = Finding.model_validate_json(res.message.content)
print(finding.severity, finding.asset, finding.action)
// bottle the house style once, reuse it forever
cat > Modelfile <<'EOF' FROM gemma4 SYSTEM """You write incident updates for an SRE team. Lead with impact, then scope, then next update time. No speculation. No apologies. Under 120 words.""" EOF ollama create -f Modelfile ollama run incident-voice
Context silently truncates. Ollama defaults to a 4096 token context window. A 400 line log will not fit. Set OLLAMA_CONTEXT_LENGTH before you trust any summary of a long file.
Binding to 0.0.0.0 has no auth. OLLAMA_HOST=0.0.0.0:11434 exposes an unauthenticated inference server. Keep it on loopback or put a reverse proxy in front of it.
Local does not mean honest. A 4B model invents CVE identifiers as confidently as anything else. Human reviews every finding before it reaches a ticket or a rule.
What Ollama actually is
Ollama is a model runner. It installs a small HTTP server on your machine, pulls open weight models from a registry, and serves them at http://localhost:11434. That is the entire product. No account needed for local models, no request leaving your network, no meter ticking while you stare at the screen and think.
For DevOps and security work that last property is the whole pitch. You can hand a model an auth log, a Terraform diff, or a scanner dump without first filing a data handling exception. The model is a process on a box you already own, under a policy you already wrote.
Ollama also offers cloud models, which are the same CLI pointed at Ollama's hosted GPUs for jobs your hardware cannot hold. That is useful, but be clear eyed: cloud models leave your box. Treat them as a different tier with different rules.
Setup in about sixty seconds
Install from ollama.com/download on macOS, Windows or Linux. On Linux the one liner is the install script. Then run ollama with no arguments for the interactive menu, or skip straight to a model.
ollama run gemma4 # pulls on first run, drops you into a chat /bye # exit the chat ollama ls # what you have on disk ollama ps # what is loaded right now
Pay attention to the PROCESSOR column in ollama ps. It reads 100% GPU, 100% CPU, or a split like 48%/52% CPU/GPU. If you picked a model too large for your VRAM, Ollama quietly spills it into system memory and your tokens per second falls off a cliff. That single column explains most "why is this so slow" complaints.
The mindset: local is a data boundary, not a benchmark
The most common way people bounce off local models is this. They install Ollama, ask a 4B model something a frontier model would nail, watch it fumble, and conclude local AI is not ready. Wrong question.
The right question is not "is this as smart as the best model in the world." It is "is this good enough for the class of work I am not permitted to send off box." That is a very different bar, and small models clear it constantly.
Sort your work into three buckets before you touch a prompt. Bucket one never leaves: production logs, customer data, incident timelines, client configs, anything under an NDA. Bucket two is public and boring: boilerplate, docs, sample code, use whatever model is best. Bucket three is heavy reasoning over sanitized input, where a cloud model earns its keep. Ollama covers buckets one and three with the same command, which is why it earns a place in the toolkit rather than replacing one.
Seven workflows worth stealing
1. Log triage that never leaves the host
Pipe straight in. Ollama reads stdin, so any log, diff or command output becomes a prompt without a copy paste round trip through a browser.
tail -n 400 /var/log/auth.log | ollama run gemma4 \ "Group these failed auth attempts by source IP. Rank by burst rate, not raw count. Flag credential stuffing patterns. Markdown table. Do not speculate about intent."
The prompt is doing real work there. Asking for grouping and ranking plays to what a small model is good at. Asking it to decide whether to block an IP does not.
2. Force JSON with a schema, not a prayer
The moment you want a model inside a pipeline, prose becomes a liability. Ollama's format field accepts a full JSON schema and constrains generation to match it, so you stop writing regex to scrape a paragraph. In Python, hand it a Pydantic model's model_json_schema() and validate the response on the way back out. Set temperature to 0 while you are at it.
One caveat straight from the docs: structured outputs are a local feature. Ollama's cloud models do not currently support them.
3. A Modelfile that bottles your team's voice
Stop pasting the same 200 word system prompt every morning. A Modelfile is four lines and gives you a named model that already knows the house style.
FROM gemma4 SYSTEM """You write incident updates for an SRE team. Lead with impact, then scope, then next update time. No speculation. No apologies. Under 120 words."""
Then ollama create -f Modelfile and ollama run incident-voice. Commit the Modelfile to the repo and the whole team gets the same behaviour. This is the single highest leverage Ollama feature that people skip.
4. Pre commit review as a git hook
A local model in a pre commit hook is fast enough to be tolerable and private enough to run against real source. Have it look for hardcoded secrets, debug flags left on, and permission changes in IaC. Keep the output advisory. A hook that blocks commits on a model's opinion will be deleted by Friday.
5. Point your coding agent at a local model
Ollama ships an ollama launch command that configures supported tools for you rather than making you hand edit config. Supported integrations include OpenCode, Claude Code, Codex, VS Code and Droid.
ollama launch # interactive picker ollama launch claude # a specific integration ollama launch claude --model qwen3.5 # pin the model ollama launch droid --config # configure, do not launch
This is the move for working on a client codebase you are contractually not allowed to send anywhere. If you want the cloud agent version of this, we covered it in our Claude Code for DevOps and security workflows walkthrough.
6. Embeddings for a runbook search you actually own
Ollama serves embedding models the same way it serves chat models. Point one at your runbooks, postmortems and internal wiki export, store the vectors in SQLite or pgvector, and you have semantic search over institutional knowledge without shipping that knowledge to a SaaS index.
ollama run embeddinggemma "Hello world" echo "Hello world" | ollama run nomic-embed-text
Retrieval also happens to be where small models shine. You are asking for similarity, not genius.
7. Cloud models when the job is genuinely too big
When you need a very large model and your box has 8 GB of VRAM, Ollama offloads to its hosted service using the same interface. Sign in once, then use a cloud tag.
ollama signin ollama run gpt-oss:120b-cloud
Say this part out loud to yourself: that request leaves your machine. Ollama states it does not store, log or train on prompts and responses for cloud models, but the data still transits. Bucket three work only. If you want the option gone entirely, set OLLAMA_NO_CLOUD=1 or put disable_ollama_cloud in ~/.ollama/server.json.
Safety and gotchas
The 4096 token default will bite you. Ollama uses a 4096 token context window unless told otherwise. Feed it a long log and the front of that log silently vanishes, and you get a confident summary of a fragment. Set it explicitly with OLLAMA_CONTEXT_LENGTH=8192 ollama serve, or per session with /set parameter num_ctx, or per request with the num_ctx option.
Binding to the network has no authentication. Ollama listens on 127.0.0.1:11434 by default for a reason. Setting OLLAMA_HOST=0.0.0.0:11434 publishes an unauthenticated inference endpoint to everything that can route to you. If you need shared access, front it with Nginx and put real auth in the proxy.
Local models hallucinate exactly as much as cloud ones. Running on your own metal changes the data boundary, not the failure mode. A small model will invent a plausible CVE identifier, a config flag that does not exist, and a package version that was never released. Every finding gets human eyes before it becomes a ticket, a rule, or a page.
Concurrency multiplies your memory bill. Required RAM scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. Turning both up because you saw them in a blog post is how a working setup starts swapping.
Cloud tags get retired, local models do not. Ollama deprecates and retires older cloud models on a published schedule, so a pinned cloud tag in a script can stop working. Models you pulled locally keep running as long as you keep the files.
Cost and hardware tips
Local inference has no per token cost, so the budget is memory and electricity. A few things that actually move the needle:
Pick the smallest model that passes your eval, then stop shopping. Write five representative prompts from your real work, run them against a 4B and an 8B, and use the smaller one if it passes. Model shopping is a very satisfying way to avoid doing the job.
Manage the load, not just the model. Models unload after five minutes idle by default. Use ollama stop to free VRAM now, or OLLAMA_KEEP_ALIVE to keep a model warm through a batch job. Preload before a batch run by sending an empty request to the API.
Quantize the cache before you downgrade the model. Setting OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves K/V cache memory against the f16 default with very little quality loss, which often buys you the headroom you were about to get by dropping to a dumber model.
Move the model directory. OLLAMA_MODELS points elsewhere. Model files are large and your system drive is not the right home for 60 GB of weights.
FAQ
Does Ollama send my prompts to ollama.com?
Not for local models. Ollama's documentation is explicit that when you run locally, prompts and data are not visible to them. Cloud models are the exception: those prompts are processed to deliver the service, and Ollama states they are not stored, logged or trained on. If your policy needs the possibility removed rather than promised, disable cloud features entirely with OLLAMA_NO_CLOUD=1 or the disable_ollama_cloud setting in ~/.ollama/server.json, and Ollama's logs will confirm it.
How much VRAM do I actually need to run Ollama usefully?
Less than people assume for the workflows above. Log summarizing, commit messages and extraction run fine on a 4B model in roughly 4 GB of VRAM. Reasoning heavy work like an incident write up wants a much larger model. Check ollama ps, and if the PROCESSOR column is not showing close to 100% GPU, drop to a smaller model rather than accepting the CPU penalty.
Can I run Ollama on a shared build server for the whole team?
You can, and it is a reasonable pattern, but do it deliberately. Ollama has no built in authentication, so exposing it with OLLAMA_HOST means anyone who can reach the port can use your GPU and read nothing back but can absolutely burn your capacity. Put a reverse proxy in front with real auth, tune OLLAMA_MAX_LOADED_MODELS, OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE to your actual memory, and monitor the queue. A 503 means the server is overloaded, not broken.
Where to take this next
Pick one bucket one task you currently do by hand because you are not allowed to paste it anywhere. Log triage is the usual first win. Get that running locally this week, wrap it in a Modelfile so the whole team gets the same behaviour, and only then look at whether a bigger model would help. Local AI pays off fastest on the work you were previously not allowed to automate at all.
If you want the structured path rather than the trial and error one, our DevOps, security and AI courses cover this end to end, and the n8n workflow guide pairs nicely if you want to schedule these local calls instead of running them by hand.
