Hetzner LLM Inference: What Bare-Metal GPU Pricing Means for Self-Hosters

The Signal: What Hetzner Actually Announced

A leaked product page and internal beta sign-up confirm that Hetzner is building a dedicated LLM Inference service. This isn’t another cloud GPU rental with a CUDA toolkit you configure yourself. It’s a managed endpoint that serves open-weight models—Llama 3, Mistral, DeepSeek variants—directly on Hetzner’s bare-metal hardware. The page lists per-token pricing, model selection, and a REST API compatible with OpenAI client libraries.

This matters because Hetzner’s entire identity is anti-cloud-markup. They sell dedicated servers at near-cost, pass through power and bandwidth with minimal margin, and have never operated a managed AI service before. Their entry into inference signals that the unit economics of serving LLMs have dropped enough that a bare-metal provider can offer competitive per-token pricing without the $3B hyperscaler overhead.

The service is currently in closed beta. You request access, they provision dedicated GPU nodes—likely their existing A100/H100 server line—and expose an endpoint. No shared tenancy, no noisy neighbors, no cold-start latency from scaling-to-zero. You’re effectively renting a dedicated inference rig with a thin API layer on top.

Why Bare-Metal GPU Inference Changes the Equation

Every major inference provider—OpenAI, Anthropic, Together, Fireworks, Groq—runs on multi-tenant infrastructure. Your prompt lands on a GPU shared with dozens of other requests, mediated by a scheduler that bounces you between warm and cold instances. This is fine for chat, but painful for production pipelines where latency variance kills SLAs.

Hetzner’s approach flips the model:

• Dedicated hardware per customer. Your inference endpoint maps to physical GPUs you’re not sharing. No token-stealing side-channels, no performance cliffs when a neighbor’s batch job kicks in.
• Predictable throughput. You know exactly how many A100s or H100s are behind your endpoint. You can calculate tokens/second deterministically.
• No egress tax. Hetzner charges flat bandwidth. If you’re streaming millions of tokens to a downstream application, you’re not paying $0.12/GB like you would on AWS or GCP.

For self-hosters who already run fine-tunes on Hetzner dedicated servers, this eliminates the "train here, serve there" split. You fine-tune on a Hetzner box, push the weights to the inference service, and serve from the same infrastructure backbone. No model weights traversing the public internet, no S3 intermediary.

The Economics: Hetzner vs. Hyperscaler Markups

Let’s run the numbers. A Hetzner dedicated server with an H100 PCIe currently costs roughly €1,500/month. An equivalent on-demand H100 instance on AWS (p4d.24xlarge, 8×A100) runs about $32/hour—over $23,000/month. Even with reserved instances, you’re paying 6-8× the bare-metal rate.

Now map that to inference pricing:

Hetzner hasn’t published final pricing, but the leaked page suggests per-token rates that undercut Together and Groq by 30-50%. This is achievable because they’re not amortizing a massive control plane, a fleet of underutilized GPUs, or a venture-scale sales org. They’re running inference directly on hardware they already own, with a thin API gateway.

For a self-hoster processing 10 million tokens/day, the monthly difference between Together ($540) and Hetzner’s projected pricing ($180–$300) pays for the server itself. At scale, you hit a crossover where owning the hardware through Hetzner’s managed layer is cheaper than renting tokens from anyone else.

Engineering Architecture: How This Fits a Self-Hosting Stack

If you’re already running inference on Hetzner boxes via vLLM, TGI, or llama.cpp, the managed service replaces the operational burden without changing your data flow. Here’s the typical self-hosting architecture and where the inference service slots in:

The key architectural win: the endpoint is OpenAI-compatible, so you swap https://api.openai.com/v1 for https://inference.hetzner.com/v1 and nothing else changes. Your existing LangChain, LlamaIndex, or custom Python client works unchanged. The difference is that behind that URL, there’s a physical GPU with your model loaded, not a virtual instance that might be paged out.

For teams running fine-tuned models, this is where Hetzner gets interesting. You train a LoRA on your dedicated server, merge the adapter, and push the resulting weights to the inference service’s attached storage. No model registry, no container build, no S3 upload. The weights live on NVMe storage colocated with the GPU, so loading a new model version is a local file operation.

How to Access and Benchmark It Today

The service is in closed beta. Here’s the practical path:

1. Request access through Hetzner’s Cloud Console. You need an existing account with billing history—they’re prioritizing customers who already run GPU workloads.
2. Specify your model. The beta supports Llama 3 (8B, 70B), Mistral (7B, Mixtral 8×7B), and DeepSeek variants. If you need a custom fine-tune, mention it in the request form.
3. Provision an endpoint. Once approved, you’ll get a dedicated endpoint URL and an API key. Provisioning takes a few minutes—they’re loading weights onto a reserved GPU, not spinning up a VM.
4. Run a benchmark suite. Don’t just fire off a few curl requests. Run a sustained load test:

import time
import httpx
from dataclasses import dataclass

@dataclass
class BenchmarkResult:
    ttft_ms: float  # time to first token
    tps: float      # tokens per second
    total_tokens: int
    latency_p50_ms: float

async def benchmark_endpoint(url: str, api_key: str, prompt: str, n_requests: int = 100) -> list[BenchmarkResult]:
    results = []
    async with httpx.AsyncClient(timeout=60) as client:
        for _ in range(n_requests):
            start = time.monotonic()
            response = await client.post(
                f"{url}/v1/completions",
                headers={"Authorization": f"Bearer {api_key}"},
                json={"model": "llama-3-70b", "prompt": prompt, "max_tokens": 256, "stream": True}
            )
            first_token = None
            token_count = 0
            async for line in response.aiter_lines():
                if line.startswith("data: ") and "[DONE]" not in line:
                    if first_token is None:
                        first_token = time.monotonic()
                    token_count += 1
            ttft = (first_token - start) * 1000 if first_token else 0
            tps = token_count / (time.monotonic() - first_token) if first_token else 0
            results.append(BenchmarkResult(ttft=ttft, tps=tps, total_tokens=token_count, latency_p50_ms=ttft))
    return results

1. Measure variance, not just mean. The whole point of dedicated hardware is consistency. If your P99 TTFT is within 15% of P50, you’re getting the bare-metal benefit. If it’s spiking 3×, something is wrong with their routing or your client’s connection pooling.

2. Compare against your current provider. Run the same benchmark against Together or Groq at the same time of day. Document the delta in both cost and latency variance.

The Cold-Start Tradeoff You Can’t Ignore

Dedicated hardware has a dark side: you’re paying for the GPU whether tokens are flowing or not. If your inference traffic is spiky—9 AM to 5 PM bursts, dead overnight—you’re burning €1,500/month on idle silicon. Serverless inference providers absorb that idle cost across their customer base; Hetzner passes it directly to you.

This makes the service a strong fit for:

• Always-on production pipelines (RAG systems, internal chatbots, CI/CD review bots)
• Latency-sensitive applications where cold-start variance breaks the user experience
• Fine-tuned models that can’t be served efficiently on shared infrastructure

And a weak fit for:

• Prototyping and experimentation where you spin up a model for an hour and tear it down
• Highly bursty workloads with 10× traffic swings
• Multi-model routing where you need 5 different models available on demand

If your workload pattern matches the weak-fit column, you’re better off with a serverless provider and eating the cold-start latency. But if you’re running a production system that’s chewing through tokens 24/7, the dedicated model saves real money.

What This Means for the Forward-Deployed Engineer

Forward Deployed Engineers sit at the intersection of infrastructure and customer outcomes. When a customer says “our RAG pipeline costs $8,000/month on OpenAI,” the FDE’s job is to find the path to $2,000/month without sacrificing latency or accuracy. Hetzner’s inference service is a new lever in that negotiation.

Here’s the FDE playbook for this capability:

1. Audit the customer’s inference spend. Pull their OpenAI/Anthropic bills. Identify which models they’re calling and at what volume. A customer doing 50M tokens/month on GPT-4o-mini is a candidate; one doing 500K tokens/month on GPT-4 isn’t.

2. Benchmark a drop-in replacement. Swap the base URL in their existing client, run the same prompts through a Hetzner-hosted Llama 3 70B, and measure quality delta. For most RAG and summarization tasks, the open model is within 5% on accuracy metrics.

3. Quantify the savings. Show them the spreadsheet: current spend vs. projected Hetzner spend, including the dedicated server cost. If the numbers work, the conversation shifts from “should we switch?” to “how fast can we migrate?”

4. Build the migration path. This is where FDEs earn their keep. Write the adapter layer, set up the monitoring, run the shadow deployment, and flip the traffic when confidence is high. The OpenAI-compatible API makes this a configuration change, not a rewrite.

This pattern—finding cost inefficiencies in a customer’s stack and replacing them with leaner infrastructure—is core to the FDE role. If you’re building this skillset, understanding GPU economics and inference serving architectures is table stakes. The FDE Compensation Bands and How to Negotiate article breaks down how infrastructure cost-saving projects directly map to performance review cycles and comp adjustments. And if you’re coming from a backend background wondering how to position yourself for this kind of work, How to Break Into FDE Roles from a Backend or Frontend Background covers the infrastructure-to-customer translation layer you need to develop.

For teams already building LLM-powered internal tools—PR review bots, invoice extractors, SQL analysts—Hetzner’s pricing changes the build-vs-buy calculus. A GitHub PR Review Bot That Comments on Logic and Style with Gemini 1.5 Flash running on a dedicated Hetzner inference endpoint could process hundreds of PRs daily at a fraction of the API cost. An Invoice Extractor That Turns PDF Receipts into Structured JSON that previously required careful token budgeting can now run on a flat monthly server cost with no per-document anxiety.

The bottom line: Hetzner entering managed inference is a signal that GPU compute is becoming a commodity. The hyperscalers will still win on ecosystem breadth, but for the narrow, high-volume use case of serving open-weight LLMs, bare-metal pricing resets the floor. Self-hosters who understand this shift can lock in infrastructure costs that make their AI features sustainably profitable, not just technically impressive.

FAQ

Q: Does Hetzner’s inference service support fine-tuned models?
Yes, but with caveats. The beta supports LoRA adapters for supported base models. Full-model fine-tunes require uploading merged weights to the dedicated NVMe storage attached to your GPU node. Expect a few minutes of downtime when swapping model versions.

Q: How does this compare to running vLLM on a Hetzner dedicated server myself?
Functionally identical, but you trade operational overhead for a thin margin. The managed service handles model loading, health checks, API compatibility, and basic monitoring. If you already have a robust vLLM deployment with Prometheus and Grafana, the managed layer adds little. If you’re a team of three without ML Ops bandwidth, it’s worth the premium.

Q: What about data privacy?
Your prompts and completions never leave the dedicated GPU node. Hetzner’s control plane sees metadata (token counts, latency) but not payload content. For regulated industries, this is a stronger privacy posture than shared-tenancy providers.

Q: Will Hetzner support GPUs beyond H100?
Almost certainly. Their server catalog already includes L40S and A100 configurations. Expect those to appear as inference options, likely at lower per-token rates for smaller models.

Q: Can I use this for training, or is it inference-only?
The managed service is inference-only. Training workloads—fine-tuning, continued pretraining—still belong on Hetzner’s unmanaged dedicated GPU servers. The inference service is the deployment target, not the training environment.