DeepSeek V4 Flash on a Single AMD MI300X: Inference Engineering Deep Dive

The Demo: What Actually Happened

On April 15, 2025, engineer Ryan Zhou dropped a GitHub repository that did something audacious: run DeepSeek V4 Flash—a 1.2 trillion parameter mixture-of-experts model—on a single AMD MI300X accelerator. Not a pod. Not a rack. A single GPU with 192 GB of HBM3 memory.

This isn't a cloud-scale demo with 8-way tensor parallelism. It's a single-process, single-GPU inference server powered by vLLM. The model is served via an OpenAI-compatible API, meaning you can point any existing toolchain at it—Open WebUI, a custom Python script, or a Slack bot—and it just works.

The raw numbers: the model weights alone, at full FP16 precision, would require over 2.4 TB of memory. The MI300X has 192 GB. The gap is closed by a combination of aggressive quantization to 4-bit precision and a carefully managed KV-cache. The result is a functional, albeit not blazing-fast, inference endpoint that can handle long-context prompts without falling over.

You can find the source repository at ryanzhou/deepseek-v4-flash-mi300x.

Why This Matters to the Inference Engineer

For forward-deployed engineers (FDEs) and anyone who builds on top of LLMs, this demo is a signal. It says: the frontier is getting portable.

Here’s the shift. Six months ago, running a model of this caliber required convincing a client to provision a multi-node cluster with InfiniBand interconnects, or burning through cloud credits on an 8xH100 instance. That meant long procurement cycles, high operational burden, and architectures that were brittle by default.

A single-GPU deployment collapses that complexity. It means:

• Air-gapped deployments become tractable. If you're deploying an LLM feature at a risk-averse enterprise, this is your wedge. No data leaves the building. The hardware footprint is a single server. We’ve covered the dynamics of this exact scenario in our case study on deploying LLM features at risk-averse enterprises.
• Prototyping without the cloud bill. You can iterate on a real, capable model locally or on a single rented bare-metal box. The feedback loop shrinks from hours to minutes.
• A new baseline for what “edge” means. Edge inference used to mean 7B parameter models. Now it means a 1.2T MoE model. The type of tasks you can do without a round-trip to a data center just changed.

This isn't just a neat trick. It's a practical unlock for the FDE weekly workflow, where speed from messy problem to shipped prototype is the only metric that matters.

The Technical Scaffolding: How It Works

The repository is a thin wrapper around vLLM, but the magic is in the configuration. Let's dissect the critical components.

The architecture is straightforward once you see it:

The Quantization Gambit

The repo uses AWQ (Activation-aware Weight Quantization) to compress the model to 4 bits per parameter. That's a 4x reduction from FP16. For a 1.2T parameter model, that brings the weight footprint from ~2.4 TB down to ~600 GB—still too large for 192 GB. So how does it fit?

Two words: sparse experts. DeepSeek V4 Flash is a mixture-of-experts model. At any given forward pass, only a fraction of the total parameters are active. The MoE architecture means the effective parameter count per token is much lower than 1.2T. Combined with 4-bit quantization, the active working set squeezes into the MI300X's memory envelope.

But there's a catch. Quantization isn't free. AWQ applies per-channel scaling factors that partially recover the accuracy lost in the 4-bit compression, but you're still trading precision for throughput. For many tasks—summarization, RAG, structured extraction—the degradation is negligible. For others, like complex multi-step reasoning, it can bite.

KV-Cache: The Silent Memory Killer

Weights are only half the story. The KV-cache stores the intermediate attention keys and values for every token in the context window. With a 128K context length, a naive KV-cache would consume tens of gigabytes on its own.

The solution here is vLLM's PagedAttention. It manages the KV-cache in non-contiguous blocks, much like an operating system manages virtual memory. This eliminates fragmentation and allows the cache to grow dynamically without pre-allocating a worst-case buffer. Combined with FP8 KV-cache storage, the memory overhead becomes manageable.

This is where the engineering gets real. You're not just loading a file; you're managing a dynamic memory system under tight constraints. One misconfigured max_model_len and the process OOMs silently.

Why AMD MI300X? The Hardware Arbitrage

The MI300X is AMD's data-center GPU with 192 GB of HBM3 and 5.3 TB/s of memory bandwidth. It's not the fastest chip on the market for AI—NVIDIA's H100 still holds the crown in raw matrix math—but it has a decisive advantage: memory capacity per dollar.

At current market rates, an MI300X gives you 192 GB of HBM3 for roughly the same cost as an H100 with 80 GB. For memory-bound inference workloads, that 2.4x capacity advantage is everything. You can run models that simply won't fit on a single H100, even with quantization.

This is hardware arbitrage in action. The dominant narrative says "buy NVIDIA." The engineer's narrative says "buy the right tool for the workload." For large-model, single-GPU inference, AMD is making a compelling case.

A Practical Guide: Running It Yourself

If you have access to an MI300X instance—through a cloud provider like Vultr, TensorWave, or a bare-metal rental—here's how to replicate the demo.

Step 1: Environment Setup

You'll need a machine with an MI300X, ROCm 6.1 or later, and Docker. The repo provides a pre-built Docker image with all the ROCm dependencies baked in.

git clone https://github.com/ryanzhou/deepseek-v4-flash-mi300x.git
cd deepseek-v4-flash-mi300x

Step 2: Download the Quantized Weights

The model is distributed via Hugging Face. You'll need a token with access to the gated repository. The download is substantial—expect several hundred gigabytes.

huggingface-cli download ryanzhou/deepseek-v4-flash-awq-4bit --local-dir ./models

Step 3: Launch the Server

The Docker Compose file configures vLLM with the correct parameters for the MI300X. Pay attention to --max-model-len—this controls the maximum context length and directly impacts memory usage.

docker compose up

Step 4: Query the API

Once the server is up, it exposes an OpenAI-compatible endpoint at http://localhost:8000/v1. You can test it with a simple curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {"role": "user", "content": "Explain the PageRank algorithm in three sentences."}
    ]
  }'

If you want a GUI, point Open WebUI at the endpoint. It's a five-minute setup for a fully functional chat interface.

For a deeper dive into building tooling around these APIs, our guide on building a SQL analyst agent with Postgres and Gemini covers patterns that translate directly to any OpenAI-compatible endpoint.

A Balanced Take: Strengths and Sharp Edges

Let's be honest about what this is and isn't.

Strengths:

• Feasibility proof. It demonstrates that frontier-scale models can run on single accelerators, which changes procurement conversations.
• AMD ecosystem maturation. The fact that vLLM, AWQ, and FlashAttention all work on ROCm without heroic effort is a big deal. The software gap is closing.
• Cost structure. If your workload is throughput-tolerant, the cost per token on an MI300X can be significantly lower than on an equivalent NVIDIA setup.

Sharp edges:

• Throughput. This is not a production serving system. Expect single-digit tokens per second. For batch processing or async workloads, fine. For interactive chat, it's borderline.
• Quantization artifacts. 4-bit models lose nuance. If you're doing precise numerical reasoning or need deterministic outputs, test rigorously before depending on it.
• ROCm rough edges. The AMD software stack has improved dramatically, but you will encounter cryptic error messages and driver issues that the NVIDIA ecosystem ironed out years ago. Budget debugging time.
• Power and cooling. The MI300X is a 750W card. A single-GPU rig is manageable, but don't expect to run this under a desk.

FAQ

Can I run this on an NVIDIA GPU instead?

The quantization technique works across hardware, but the memory constraint is the bottleneck. You'd need an H100 NVL with 188 GB to even attempt it, and those are rare. The MI300X's 192 GB is the key enabler here.

What's the actual tokens-per-second performance?

The repo author reports roughly 5-8 tokens/second for typical prompts. This varies significantly with context length—longer contexts slow generation as the KV-cache grows.

Is the model quality noticeably degraded at 4-bit?

For general-purpose chat, summarization, and RAG tasks, the degradation is often imperceptible. For tasks requiring precise recall of training data facts or complex mathematical reasoning, you may notice a drop. Always benchmark against your specific use case.

Why not use a smaller model at full precision?

You could, and for many use cases that's the right call. But MoE models have a unique property: they maintain broad world knowledge and strong reasoning capabilities even at aggressive quantization levels, because the expert routing provides a form of implicit regularization. A dense 70B model at FP16 might beat a quantized 1.2T MoE on some benchmarks, but the MoE will win on breadth.

What's the next step for this approach?

Expect to see FP6 and FP4 quantization schemes mature, along with speculative decoding to claw back throughput. The combination could make single-GPU frontier inference genuinely practical for production.