Running LLMs Locally in Logistics: Ollama + VPN + Air-Gapped Deployment

I've burned more GPU budget on "private cloud" demos than I care to admit. At AINNA, where we run NeuralOps infrastructure and a growing fleet of Detached Systems, one of the hardest problems we solve is putting Large Language Models inside logistics environments that cannot - and should not - phone home to OpenAI. This article is the technical blueprint I wish I had three years ago: Ollama + VPN + air-gapped deployment, built for real 3PL warehouses, freight brokers, and Malaysian SMEs that handle customer manifests, customs forms, and supplier contracts.

1. Why Logistics Needs Local LLMs

Logistics data is messy, regulated, and high velocity. A typical 3PL in Melaka processes thousands of airway bills, packing lists, and customs declarations every day. Most of those documents contain:

• Customer names, IC numbers, and phone numbers (PDPA-sensitive).
• Supplier pricing and contract terms (commercially sensitive).
• Customs HS codes, declared values, and origin data (legally sensitive).

Sending that to a public API - even with promises of "zero retention" - is a non-starter for many compliance officers I've worked with. Local inference removes that risk. But it also introduces new ones: model management, GPU failures, context-window crashes, and VPN latency that can make a chatbot feel like it's typing on a typewriter.

We originally tried a hybrid approach: sensitive extraction on local models, summarization on Groq. It worked until a customs broker asked, "Where exactly did my Bill of Lading go?" That single audit question killed the hybrid design. Now our default architecture at AINNA is VPN-first, local-first. Cloud LLMs are an explicit opt-in, not a default.

2. The Stack: Ollama, WireGuard, and Air-Gapped vLLM

2.1 Ollama as the Edge Inference Engine

Ollama is not the fastest inference engine. It is not the most feature-complete. What it is, in my experience, is the most reliable way to get a model running on an edge box in under ten minutes. For logistics clients who need to deploy in a warehouse with no dedicated ML engineer on site, that matters more than raw throughput.

Our standard edge node is a workstation with an NVIDIA RTX A6000 (48 GB VRAM), 128 GB RAM, and a 2 TB NVMe. That box lives in the warehouse rack, behind the client's firewall, reachable only over the AINNA-managed WireGuard VPN. Ollama runs inside Docker with a restricted Modelfile tuned for structured extraction:

# /models/qwen-logistics/Modelfile
FROM qwen2.5-coder:14b
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 1024
PARAMETER top_p 0.3
PARAMETER repeat_penalty 1.1
SYSTEM "You are a logistics document parser. Extract only the fields requested. Respond in valid JSON. Do not add explanations."

Why qwen2.5-coder:14b? In our benchmarks it beats llama3.1:8b on structured JSON extraction from mixed English/Malay/Bahasa documents, and it fits comfortably in 48 GB with headroom for the KV cache. The 14B parameter size is the sweet spot for edge: good enough accuracy, small enough latency, cheap enough hardware.

2.2 VPN-First, Not Cloud-First

Every AINNA deployment starts with a WireGuard hub-and-spoke topology. The hub is a hardened VPS or the client's own NeuralOps Cloud Managed node. The spokes are the warehouse edge boxes, finance office desktops, and any detached inventory servers. No spoke has a public IP. No spoke can reach the internet directly. All control-plane traffic - model pulls, telemetry, agent orchestration - flows through the hub.

# /etc/wireguard/wg0.conf on a warehouse spoke
[Interface]
PrivateKey = 
Address = 10.200.0.5/32
ListenPort = 51820
DNS = 10.200.0.1

[Peer]
PublicKey = 
AllowedIPs = 10.200.0.1/32, 10.201.0.0/24
Endpoint = neuralops-hub.ainna.bond:51820
PersistentKeepalive = 25

The AllowedIPs line is the critical security control. It means the warehouse Ollama node can talk to the NeuralOps hub and the internal analytics subnet, but not to Google, GitHub, or a rogue mirror. When we need to update a model, we stage the GGUF or safetensors on the hub, sign it, and push it down through the VPN. The edge node never downloads from the public internet.

2.3 Air-Gapped vLLM Cluster for Heavy Workloads

Ollama is great for single-box inference. For a centralized cluster serving multiple warehouses - what we call the AI Agent & LLM Server Layer in NeuralOps - we use vLLM. A typical hub cluster is two A100 80GB GPUs in tensor-parallel mode, running llama-3.3-70b AWQ-quantized:

python -m vllm.entrypoints.openai.api_server \
  --model /models/Llama-3.3-70B-Instruct-AWQ-INT4 \
  --served-model-name llama-3.3-70b-logistics \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 32768 \
  --max-num-batched-tokens 8192 \
  --port 8000 \
  --api-key ${VLLM_API_KEY}

This cluster has no outbound NAT. Model weights live on an encrypted NFS mount. Logs ship to a local Loki instance, never to a SaaS. Backups go to a tape rotation. It is, by design, boring infrastructure - and boring infrastructure is what keeps auditors happy.

3. Model Selection for Logistics Workloads

We do not deploy one model and hope. We run a model router in the AI Agent Layer that picks the cheapest capable model for each task. Here is what actually works in production:

3.1 Coding and Agentic Tasks

• kimi-k2.7-code:cloud: We use this for complex agent reasoning when the client permits cloud usage. It is expensive but excellent at multi-step planning - e.g., "Reschedule this container shipment around a port strike and update the inventory reservations." On a cloud endpoint we see roughly $0.02 / 1K tokens depending on provider. We only route high-value decisions here.
• qwen2.5-coder: Our default on-premise code and extraction model. Handles JSON, SQL generation for the Inventory System, and light report automation. Runs at ~120 ms/token on the A6000 edge node.

3.2 Reasoning and Document Parsing

• deepseek-coder: Strong for long-context document understanding. We use it for customs declaration checks where the prompt includes 20+ pages of HS-code reference tables. Watch out: the 236B variant needs serious VRAM. We usually run the 33B distilled version locally.
• llama-3.3-70b: The "safe choice" for mixed English/Bahasa conversations in the AI Agent Dashboard. Quantized to AWQ INT4, it fits on two A100s and gives acceptable quality for chat and classification. Not the cheapest to run locally - power and cooling are real costs - but cheaper than leaking data.

Cost reality check: A 1,000-token call to a cloud API might cost $0.02 on Groq or $0.03+ on OpenAI. The same call on local Ollama costs $0 in API fees - but you already paid RM 50,000 for the server, RM 800/month for power, and my team's time to keep it patched. Local is not free. It is a capex-and-compliance trade, not a price hack.

4. AINNA's VPN-First, Hub-and-Spoke vLLM Architecture

Here is the pattern we repeat across Malaysian logistics deployments. I call it "hub-and-spoke vLLM."

1. Hub: One or more GPU servers in the client's data center or a dedicated NeuralOps Cloud AI/Agent node. Runs vLLM, the model registry, the agent orchestrator, and the central vector database.
2. Spokes: Warehouse edge nodes running Ollama for low-latency extraction and offline fallback. Each spoke connects to the hub via WireGuard. Spokes can operate autonomously if the VPN drops.
3. Control plane: AINNA NeuralOps manages WireGuard keys, Docker image updates, database backups, and 24/7 monitoring. We do not manage the client's network - we secure the overlay.

Client applications talk to the models through a thin OpenAI-compatible client. The base URL points to a VPN address, not a public endpoint:

import openai
import os

client = openai.OpenAI(
    base_url="http://10.201.0.5:11434/v1",
    api_key="ollama",
    timeout=30.0
)

resp = client.chat.completions.create(
    model="qwen2.5-coder:14b",
    messages=[
        {"role": "system", "content": "Extract consignee, HS code, and declared value as JSON."},
        {"role": "user", "content": doc_text[:6000]}
    ],
    max_tokens=512,
    temperature=0.0,
    response_format={"type": "json_object"}
)
print(resp.choices[0].message.content)

That base_url is reachable only inside the WireGuard tunnel. If an attacker compromises the warehouse LAN, they still cannot reach the model server unless they also have the WireGuard private key and the hub authorizes their IP. Defense in depth is not a slogan; it is the only reason I sleep on Friday nights.

5. Token Optimization and Latency Tuning

Local inference saves money only if you are not wasting tokens. I have seen warehouses burn 30% of their GPU capacity on bloated prompts and oversized context windows.

5.1 Context Window Budgeting

Do not set num_ctx = 128k because the model supports it. KV cache scales quadratically with context length in attention, and on consumer GPUs that translates directly to slower generation and higher VRAM use. For airway bill extraction, we cap context at 8,192 tokens. For long-form customs analysis, we chunk documents into 4,000-token windows and run parallel inference, then merge results.

5.2 Batching and KV Cache

vLLM's PagedAttention is the reason we use it for the hub. We configure max-num-batched-tokens and max-num-seqs based on observed traffic, not defaults. For a 3PL with 40 concurrent agents, we typically run:

--max-num-seqs 64
--max-num-batched-tokens 16384
--gpu-memory-utilization 0.90

That pushes the A100s to ~85% utilization without OOM crashes. Ollama does not support continuous batching the same way, so for edge nodes we limit concurrent requests in the reverse proxy (usually Nginx or Caddy) and queue the rest.

5.3 Quantization Trade-offs

We quantize almost everything that runs locally. AWQ INT4 and Q4_K_M are standard. The quality drop is measurable on coding tasks but rarely on logistics extraction, where the output schema constrains the model heavily. My rule of thumb: if the task is classification or structured extraction, quantize aggressively. If the task is open-ended reasoning or customer-facing chat, keep more bits.

6. Production Failures and What Actually Works

I will not pretend this has been smooth. Here are three failures that shaped our current architecture.

6.1 The 32K Context Crash

We deployed llama-3.3-70b with max-model-len 32768 and told the finance team they could dump entire annual ledgers into the Finance Dashboard. It worked in testing. In production, the first user pasted a 28,000-token spreadsheet and the vLLM process was killed by the OOM killer. Root cause: long context plus high batch size exceeded VRAM. Fix: cap per-request context to 16K, add input length validation in the API gateway, and expose a token counter in the UI so users see the cost before they submit.

6.2 VPN Latency Surprises

A client in Johor routed all traffic through a hub in Kuala Lumpur because "the cloud is in KL." The round-trip added 35 ms. That does not sound like much, but LLM inference is a sequential token stream. For a 500-token response, 35 ms per token becomes 17.5 seconds of perceived latency. We moved the hub closer - same city, same ISP - and latency dropped below 4 ms per token. Lesson: place inference where the users are, not where the cloud provider is.

6.3 Model Drift on Air-Gapped Nodes

Edge nodes that never touch the internet also never get silent security patches. We once had a spoke running an outdated Ollama version with a known container escape. Because it was air-gapped, the CVE was theoretical - until a technician plugged in a USB drive. Now every detached system has an offline patch schedule: signed update packages staged on the hub, pushed over VPN, installed during maintenance windows, and verified by AINNA's monitoring layer before the node rejoins the mesh.

7. Security, PDPA, and Malaysian SME Reality

Malaysia's Personal Data Protection Act 2010 (PDPA) does not explicitly ban cloud AI, but it requires clear purpose limitation, consent, and security safeguards. In practice, that means a DPO asking uncomfortable questions about data residency. Local LLMs give a clean answer: data never leaves the premises.

We layer the usual NeuralOps controls on top:

• SSL/TLS Encryption inside the VPN, because encryption should not stop at the tunnel.
• Docker Isolation for Ollama and the AI Agent Dashboard.
• Comprehensive Logging to a local SIEM, not a foreign SaaS.
• Automated Backup of model configs, fine-tunes, and vector stores.
• 24/7 Monitoring with alerts for GPU temperature, OOM events, and unauthorized API calls.

For a Melaka SME running a single 3PL warehouse, the total cost of a local Ollama deployment - server, switch, UPS, VPN setup, and managed support - is roughly RM 45,000–65,000 first year. That is not cheap. But a single PDPA fine or a leaked supplier contract can cost far more. We do not sell local LLMs as a cost saver. We sell them as a risk reducer.

8. Bottom Line

Running LLMs locally in logistics is not a hobbyist project. It is an infrastructure decision with real trade-offs in cost, latency, security, and model quality. My production-tested recipe is:

• Use Ollama on warehouse edge nodes for fast, offline, structured extraction.
• Use vLLM on a central GPU hub for heavy reasoning and multi-tenant agents.
• Connect everything with a WireGuard VPN hub-and-spoke; never expose model APIs to the public internet.
• Keep the cluster air-gapped by default, with signed, staged updates.
• Pick models pragmatically: qwen2.5-coder for edge coding/extraction, deepseek-coder for long documents, llama-3.3-70b for general chat, and cloud models like kimi-k2.7-code:cloud only when the business case justifies the risk.
• Optimize tokens, context windows, and batch sizes like you optimize SQL queries - because wasted tokens are wasted GPU dollars.

At AINNA, this pattern powers our Detached Systems, Ecommerce Systems, Inventory Systems, and the AI Agent Dashboards we ship to logistics clients across Malaysia. It is not the flashiest architecture. It is the one that still works at 2 AM when a customs officer asks for an audit trail and the internet is down.

That is why I build local-first.