MI355X vs B300 for MoE Inference: Kimi K3 Throughput per Dollar Deep Dive

The Benchmark: What Actually Happened

Wafer.ai ran a direct, apples-to-apples inference benchmark that feels like a shot across the bow. They took Kimi K3, a massive Mixture-of-Experts (MoE) language model from Moonshot AI, and ran it on two different datacenter GPUs: the AMD Instinct MI355X and the Nvidia B300 (Blackwell). The result? The MI355X delivered better performance per dollar.

This isn't a synthetic benchmark. It's a real-world inference test on a production-grade MoE model with over 600 billion total parameters (around 16 experts, with 2-4 active per token). The metric that matters here is throughput per dollar—how many tokens you can push through the system for every buck you spend on hardware.

The raw numbers tell the story. The MI355X, with its 288 GB of HBM3e memory and 8 TB/s of memory bandwidth, excels at the memory-bound workloads that define large MoE inference. The B300 is no slouch, but its higher price tag shifts the cost-efficiency equation. When you're serving millions of tokens, the difference in $/token adds up to real money, fast.

You can read the full breakdown on the Wafer.ai blog.

Why This Matters: An Engineer's Take on the Silicon Shift

For the working engineer—especially a Forward Deployed Engineer (FDE) who lives at the intersection of cost, performance, and customer reality—this benchmark is a signal flare. Three things are happening at once:

1. The Inference Cost Crunch is Real. We're past the era of just making models bigger. The new battleground is cost-efficient serving. If you're building a RAG pipeline, an agentic workflow, or a customer-facing chat product, your COGS (Cost of Goods Sold) is dominated by inference spend. A 20-40% improvement in $/token isn't a rounding error; it's the difference between a viable product and a money pit.
2. Hardware Diversity is No Longer Theoretical. For a decade, "just buy more Nvidia" was the only answer. That's changing. If you're an FDE scoping a prototype that needs to scale, you now have a credible second source. This changes vendor conversations, architecture decisions, and your own mental model of what's possible. It's the same feeling you get when moving from a messy customer problem to a shipped prototype in a week—constraints are shifting, and you have new leverage.
3. MoE Architectures Change the Hardware Calculus. A dense model's performance is often compute-bound. An MoE model, especially one as large as Kimi K3, is brutally memory-bound. This flips the script on which GPU specs actually matter.

The Architecture of the Problem: Memory Bandwidth is King

To understand why the MI355X wins on cost, you have to look at what an MoE model actually does during inference. This isn't a dense transformer where the entire model is activated for every token. In an MoE model like Kimi K3:

• Total parameters might be 600B+, but only a fraction (e.g., 2 out of 16 experts) are active for any given token.
• This means the active parameter count per token might be around 75B.
• However, the entire model still needs to be resident in memory. You can't just load the two experts you need next; the routing decision happens dynamically.

This creates a massive memory capacity and bandwidth bottleneck. Every single token generation step requires reading the weights of the active experts from HBM into the compute units. With 75B active parameters at FP8 (1 byte per parameter), you're shuffling 75 GB of data per token just for the weights, plus KV cache overhead.

Here's a simplified flow of what happens during a single decoding step:

The compute units are often starved, waiting for weights to arrive. This is why memory bandwidth is the single most critical spec for MoE inference throughput.

Let's look at the hardware comparison:

Notice the brutal truth of this table. The B300 has nearly double the raw compute. But for an MoE model, that compute is wasted because the chip is memory-bandwidth-bound. Both cards have similar memory bandwidth, but the MI355X has 50% more capacity and costs less. The math is straightforward: you get roughly the same tokens per second, but you pay less for the hardware, and you can fit a larger model or a bigger KV cache on a single device.

This is the same fundamental tradeoff we explored when running Kimi K3 locally on consumer hardware—the memory wall is the great equalizer.

A Practical Guide: Trying Kimi K3 on AMD Silicon Today

You're an engineer. You want to get your hands dirty. Here's the current state of play for actually running an MoE model like Kimi K3 on AMD hardware.

The Direct Route: vLLM with ROCm

The most practical path is using vLLM, which has rapidly matured its AMD support via the ROCm ecosystem. The process mirrors an Nvidia setup, but with a different Docker base image.

# Pull the ROCm-enabled vLLM image
docker pull rocm/vllm:latest

# Run the server with an MoE model (example with a DeepSeek-V2 variant)
docker run -it \
  --network host \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /path/to/models:/models \
  rocm/vllm:latest \
  python -m vllm.entrypoints.openai.api_server \
    --model /models/Kimi-K3-Instruct \
    --tensor-parallel-size 8 \
    --dtype bfloat16

The Abstraction Route: Cloud APIs

If you don't have a rack of MI355Xs lying around, you'll access this hardware through a cloud provider. The workflow is identical to what you do today. You change an API endpoint and an API key, and the underlying silicon becomes someone else's problem. This is where the $/token metric becomes your procurement argument.

The FDE Angle: Prototyping with Cost Leverage

When you're scaling yourself and deciding when to hand off a prototype, hardware cost is often the silent killer of a project. A prototype that costs $500/day in inference to test with a few customers is a non-starter. A prototype that costs $200/day on alternative silicon suddenly has room to breathe. This benchmark gives you the data to make that case. You can build a local codebase Q&A tool or a complex agent and, when it's time to scale the model, have a clear-eyed conversation about hardware economics that isn't just "we need more A100s."

The Balanced View: Software, Scale, and the CUDA Moat

Let's not get carried away. A single benchmark does not a market shift make. The Nvidia ecosystem advantage remains massive, and it's built on three pillars:

1. CUDA Moat: It's not just about writing kernels. It's about the debuggers, profilers, libraries (cuDNN, cuBLAS, TensorRT-LLM), and the collective muscle memory of millions of engineers. ROCm has made incredible strides, but you will hit edge cases. A custom fused kernel that takes an afternoon to write in CUDA might take a week of fighting with HIP. For an FDE debugging in a customer's environment without direct access, the last thing you need is a platform-level variable.
2. Networking and Scale-Out: The benchmark likely ran on a single node. Real-world MoE inference often requires sharding the model across multiple GPUs. Nvidia's NVLink and InfiniBand ecosystem is a known quantity. AMD's Infinity Fabric is powerful, but the software and reference architectures for large-scale MoE inference are less battle-tested. If your deployment needs 16 GPUs tightly coupled, the integration risk is higher on the AMD side.
3. Availability and Supply: The B300 is the latest thing. The MI355X is also new. Actually getting your hands on either in volume is a supply chain negotiation, not a commodity purchase. The theoretical cost advantage only matters if you can get the hardware.

The Bottom Line: This isn't a "winner takes all" moment. It's the opening of a real competition. For engineers, that's unequivocally good news. It means you should start treating your inference stack like you treat your cloud provider—abstracting away the hardware specifics where possible, and maintaining a clear-eyed view of $/token as the ultimate metric you own.

FAQ: MI355X, B300, and MoE Inference

Q: Does this mean AMD is faster than Nvidia?

A: Not universally. In this specific, memory-bandwidth-bound MoE inference workload, the MI355X delivers better performance per dollar. The B300 has a raw compute advantage that shines in compute-bound tasks like training or dense model inference. The key is matching the workload to the hardware strength.

Q: Can I run Kimi K3 on a single MI355X?

A: Kimi K3's full 600B+ parameters, even at FP8, require roughly 600 GB of memory just for the weights. With 288 GB of HBM, a single MI355X cannot hold the entire model. You would need to shard the model across at least 2-3 MI355Xs using tensor parallelism, or use a heavily quantized version. The benchmark from Wafer.ai almost certainly used a multi-GPU setup.

Q: What's the single most important spec for MoE inference?

A: Memory bandwidth, followed closely by memory capacity. Compute (FLOPS) is a distant third. You are fundamentally moving weights from memory to compute, and the compute sits idle most of the time. This is why the MI355X's 8 TB/s bandwidth and 288 GB capacity make it so competitive.

Q: Should I rewrite all my CUDA code for ROCm now?

A: No. The right move is to push for abstraction layers. Use frameworks like vLLM, PyTorch, or Triton that compile to multiple backends. Let the framework developers fight the ROCm/CUDA battle. Your job is to write PyTorch models and select a device string. This keeps your codebase portable, which is the real strategic win.

Q: How do I measure $/token for my own project?

A: Don't overcomplicate it. Total cost of hardware (amortized over 3-5 years) + electricity + cooling, divided by total tokens served in that period. For a prototype, use cloud on-demand pricing for the GPU instance as a proxy. Track it religiously. It's one of the metrics an FDE actually owns because it directly correlates with whether a feature is economically viable.