← Blog
#LLM#AI#Self-hosting#Llama#Infrastructure

Local LLMs Are Production-Ready: API vs Self-Hosting in 2026

webhani·

The Register published a piece in May 2026 titled "Yes, local LLMs are ready to ease the compute strain." It's worth taking seriously. While frontier API models like Claude Opus 4.7 and GPT-5.5 get most of the attention, open-weight models have quietly crossed a quality threshold that makes self-hosting viable for production workloads.

This post covers when to self-host, how to set it up, and the tradeoffs you'll actually care about.

Why the Calculus Has Changed

Until mid-2025, self-hosting meant accepting significant quality compromises. That's less true now.

Llama 4 Scout (Meta's edge-inference-optimized model) and Qwen 3.7 Max (released at Alibaba Cloud Summit on May 20, 2026, scoring 56.6 on the Intelligence Index) deliver results that meet the bar for most enterprise tasks — text classification, summarization, code review, structured extraction, and basic question-answering.

Three factors make self-hosting worth evaluating:

Cost at scale: API pricing is per-token. High-volume workflows — automated code review, document indexing, customer support triage — can generate surprising monthly bills. Self-hosting trades upfront GPU cost (or cloud GPU time) for predictable compute.

Latency: API round-trips typically take 200ms–2s. For real-time UX or tight processing pipelines, eliminating network hops matters.

Data locality: Some organizations can't send code, contracts, or customer data to a third-party API. Local inference solves this cleanly.

Getting Started with Ollama

Ollama is the simplest path to a local inference server. It runs on macOS, Linux, and Windows and exposes an OpenAI-compatible REST API.

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh
 
# Pull and run Llama 4 Scout
ollama run llama4:scout
 
# The inference server starts at http://localhost:11434
ollama serve

Because Ollama exposes an OpenAI-compatible endpoint, existing SDK code only needs a base URL change:

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any non-empty string works
)
 
response = client.chat.completions.create(
    model="llama4:scout",
    messages=[
        {"role": "user", "content": "Review this code for potential issues."}
    ],
)
print(response.choices[0].message.content)

Scaling on Kubernetes

For production deployments beyond a single machine, Kubernetes with NVIDIA GPU Operator is the standard approach:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ollama
  template:
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          resources:
            limits:
              nvidia.com/gpu: 1
          ports:
            - containerPort: 11434
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models-pvc

Cache model weights in a PersistentVolume shared across pods to avoid re-downloading multi-GB files on every pod restart.

Hybrid Routing

You don't have to make a binary choice. Routing requests based on task complexity works well in practice:

async def invoke_llm(prompt: str, task_type: str = "routine") -> str:
    if task_type == "complex_reasoning":
        # Hard problems get the frontier API model
        return await call_claude_opus(prompt)
    else:
        # Routine work stays local
        return await call_local_llama(prompt)

This pattern keeps API costs bounded while ensuring the highest quality where it matters most.

Decision Criteria

Choose the API when:

  • You need frontier-tier quality (Claude Opus 4.7, GPT-5.5-class performance)
  • GPU infrastructure and ops overhead aren't worth it for your scale
  • Usage patterns are unpredictable
  • You're prototyping

Choose self-hosting when:

  • Monthly token volume is high enough that API bills are a real concern
  • Data can't leave your infrastructure (compliance, contractual, or legal requirements)
  • You need sub-100ms inference
  • You're operating in environments with unreliable or no internet connectivity

Operational Considerations

Self-hosting comes with real ops responsibilities:

  • Model versioning: Define a process for updating to new model releases without disrupting running services
  • GPU failure handling: Implement fallback to the API if GPU nodes become unavailable
  • Observability: Track inference latency, GPU utilization, error rates, and queue depth
  • Quality monitoring: Run task-specific eval benchmarks periodically — model updates can shift behavior in unexpected ways

Takeaway

For organizations already spending significantly on LLM APIs, a self-hosting pilot is worth the evaluation time. Ollama gets you running in under an hour, and the OpenAI SDK compatibility means minimal application changes. Start with an internal workflow — CI code review, documentation generation, support ticket classification — measure quality and cost, then expand from there.

The open-weight models of 2026 aren't "good enough for demos." They're good enough for production.

← Back to Blog