Framework Desktop Packs 192 GB RAM & AMD Ryzen AI Max+ Pro 495 for Local LLMs

What Just Dropped: The 192 GB Framework Desktop

Framework, the modular laptop company that proved repairable hardware isn’t a fantasy, has done something unexpected. They’ve taken AMD’s new Ryzen AI Max+ Pro 495—a chip built for mobile workstations—and strapped it to a desktop mainboard with a staggering 192 GB of unified LPDDR5x memory. This isn’t a laptop with a big RAM config. It’s a dedicated desktop tile, shipping inside Framework’s existing Desktop case (or as a standalone board for the BYO-crowd), and it’s labeled “Coming Soon” on their site.

The CPU itself is a 16-core/32-thread Zen 5 part with an integrated RDNA 3.5 GPU packing 40 compute units. But the real headline is the memory controller. AMD’s “AI Max” branding isn’t just marketing fluff—this chip treats the entire 192 GB pool as unified memory, meaning the CPU and the iGPU share the same physical address space with zero copy overhead. For anyone who’s ever watched an llama.cpp load scream data back and forth between system RAM and a discrete GPU’s VRAM over PCIe, you know exactly why this is interesting.

Framework is positioning this as a local AI workstation. No discrete GPU required. No exotic cooling. Just a single board, a power supply, and a dream. The source is Framework’s own Desktop page, where you can stare at the “192 GB Coming Soon” tab and wonder how many quantized 70B models you could fit in there. (Answer: all of them, simultaneously, with room for a codebase index.)

Why Unified Memory Architecture Matters for Local Models

To understand why this board is a bigger deal than “just another mini PC,” you need to understand the memory wall that local inference keeps hitting.

In a traditional desktop with a discrete GPU, you have two memory pools: system RAM (DDR5, cheap, 64-128 GB typical) and VRAM (GDDR6/6X, fast, 12-24 GB typical on consumer cards). When you load a model, you have to decide where it lives. If it fits in VRAM, great—you get full memory bandwidth and low latency. If it doesn’t, you spill into system RAM and the GPU has to page data across the PCIe bus. That bus runs at roughly 32-64 GB/s in practice, which is an order of magnitude slower than the 800-1000 GB/s that modern GDDR6X delivers. Your inference speed tanks. You start using quantized models (Q4, Q3, Q2) to squeeze things down, and you lose precision.

A unified memory architecture (UMA) like this one collapses that problem. There is one pool. The CPU and GPU cores see the same physical memory. No copies. No PCIe transfers. No cudaMemcpy calls silently eating your token generation speed. The entire 192 GB is available to the GPU cores at the memory controller’s native bandwidth.

For a forward deployed engineer who needs to run a local RAG pipeline, a fine-tuned 70B model, or an agent that chains multiple models together, this means you can load a full-precision 70B model (roughly 140 GB at FP16) entirely in a single memory space and still have 50 GB left for context, embeddings, and your operating system. You can run Llama 3.1 405B at aggressive quantization without splitting across machines. You can serve a model locally for testing before it touches a cloud endpoint.

This is the first consumer-adjacent hardware that makes local large-model inference feel like a solved problem, not a compromise.

The Spec Sheet That Actually Matters to an Engineer

Let’s cut through the marketing and look at the numbers that affect your workloads.

That power number is worth pausing on. 120 watts. For a machine that can hold 192 GB of model weights in a unified pool. Compare that to a dual-3090 rig pulling 700W from the wall just to get 48 GB of VRAM. The efficiency here is obscene.

The 256-bit memory bus width is the spec that tells you the most about what this machine actually is. AMD didn’t slap a 512-bit or 1024-bit interface on this. It’s a mobile-derived memory controller. That means the bandwidth ceiling is real, and it’s lower than what you’d get from a stack of HBM on a datacenter GPU. We’ll get into why that matters.

How to Use This Rig for Local Inference Today

If you get your hands on this board, here’s the practical path to running models on it.

Step 1: OS and Driver Setup

You’ll want a Linux distribution with recent kernel support for AMD’s iGPU compute stack. Ubuntu 24.04 or Fedora 40+ are safe bets. Install the ROCm stack for the RDNA 3.5 GPU—this is what gives you GPU-accelerated inference on AMD silicon. At the time of writing, ROCm 6.1+ supports RDNA 3.5. You’ll also want the AMDGPU kernel driver, which should load automatically on a modern kernel.

Step 2: Choose Your Inference Engine

The two main paths are llama.cpp with Vulkan or ROCm backends, and vLLM or LM Studio if you want a server-style setup. For this hardware, llama.cpp with the ROCm backend is likely the sweet spot. It’s memory-efficient, supports every quantization format you’d want, and doesn’t require the CUDA ecosystem.

# Build llama.cpp with ROCm support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_HIPBLAS=1 -j

Step 3: Load a Large Model

With 192 GB, you can run:

• Llama 3.1 70B at FP16: ~140 GB, fits with 50 GB to spare for KV cache and system overhead
• Mixtral 8x22B at FP16: ~280 GB, won’t fit at full precision but runs comfortably at Q4_K_M (~80 GB)
• Llama 3.1 405B at Q4_K_M: ~230 GB, tight but possible with minimal context
• Command R+ (104B) at FP16: ~208 GB, borderline but doable with aggressive memory management

For real workloads, you’re not just loading weights—you’re allocating KV cache for context. A 128K context window on a 70B model can eat 30-50 GB of additional memory. Plan accordingly.

Step 4: Run an Agent or RAG Pipeline

This is where the unified memory shines. You can run the LLM, the embedding model, and the reranker all on the same machine without model swapping. For example, load llama-3.1-70b-q4_K_M as your generator, bge-large-en-v1.5 for embeddings, and a small cross-encoder for reranking—all resident simultaneously. Your agent loop calls each model without a cold start.

If you’re building something like a SQL analyst agent over Postgres or a PR review bot with code context, this hardware lets you test the full pipeline locally before deploying to a cloud GPU. No more “it works on my Colab notebook but dies on the server” moments.

The Hidden Constraint: Memory Bandwidth, Not Just Capacity

Here’s the part that separates the engineers who read spec sheets from the ones who just see “192 GB” and get excited.

Memory capacity is half the story. Memory bandwidth is what determines your tokens-per-second. The Ryzen AI Max+ Pro 495 uses a 256-bit LPDDR5x interface. At LPDDR5x-7500 speeds, that gives you roughly 240-256 GB/s of theoretical bandwidth. In practice, you’ll see 200-220 GB/s after overhead.

What does that mean for inference speed?

A 70B parameter model at Q4_K_M quantization is about 40 GB of weights. To generate one token, you need to read every weight once (assuming a memory-bandwidth-bound decode phase). At 220 GB/s, that’s 40 GB / 220 GB/s = 0.18 seconds per token, or about 5.5 tokens per second. That’s usable. It’s not fast, but it’s conversational.

The same model at FP16 (140 GB) would be 140 / 220 = 0.64 seconds per token, or about 1.6 tokens per second. That’s slow. You’d feel it in a chat interface.

Compare this to an NVIDIA RTX 4090 with 1008 GB/s of memory bandwidth. A 70B Q4 model that fits entirely in its 24 GB VRAM would run at roughly 25 tokens per second. But the 4090 can’t hold a 70B model at Q4—it’s too big. You’d need two 4090s or an A6000. The Framework board sacrifices raw speed for sheer capacity.

The engineering tradeoff is clear: this is a capacity play, not a speed play. If you need to run a model that’s too big for any single consumer GPU, and you don’t want to mess with multi-GPU setups or cloud rental, this board is your answer. If you need 50+ tokens per second on a 7B model, buy a 4090.

For FDE workflows—where you’re prototyping agents, running evals, and testing pipelines—5-10 tokens per second on a 70B model is perfectly fine. You’re not serving production traffic. You’re iterating.

Where It Wins and Where It Still Loses

Let’s be honest about what this machine is and isn’t.

Wins

• Model capacity per dollar: 192 GB of unified memory in a single socket at a consumer price point (Framework hasn’t announced pricing, but expect $2,000-3,000 based on their existing Desktop pricing and the cost of LPDDR5x) is unprecedented. A Mac Studio with 192 GB runs $5,599+.
• Power efficiency: 120W for 192 GB of model capacity. Your electricity bill and cooling setup will thank you.
• No PCIe bottlenecks: The unified memory model eliminates the biggest pain point in multi-GPU inference rigs.
• Modularity and repairability: It’s Framework. You can swap the board, the storage, the expansion cards. No soldered-down nonsense.
• Local agent development: Perfect for building and testing the kinds of multi-model pipelines that FDEs ship—think invoice extraction with vision LLMs or Gmail triage agents that chain classification, extraction, and drafting.

Losses

• Memory bandwidth ceiling: 256 GB/s is an order of magnitude below a datacenter GPU. You won’t be serving production traffic on this.
• No CUDA: The ROCm ecosystem has improved dramatically, but it’s still not CUDA. Some libraries and tooling assume NVIDIA hardware. You’ll hit edge cases.
• Integrated GPU, not discrete: The RDNA 3.5 iGPU is capable but it’s not an H100. Compute-bound operations (prompt processing, speculative decoding) will be slower than on a discrete GPU.
• Single socket: No multi-node scaling. This is a workstation, not a cluster node.
• AMD’s software maturity: ROCm on RDNA 3.5 is still relatively new. Expect driver quirks and the occasional need to compile from source.

The Verdict

This is the best local inference machine for engineers who need to run large models but don’t want to build a multi-GPU space heater. It’s not a production server. It’s a development workstation that happens to have enough memory to hold a 405B model at low precision. If your workflow involves iterating on prompts, testing agent architectures, or running local evals on full-size models, this is the hardware you’ve been waiting for.

If you’re an FDE who needs to understand what metrics actually matter or how to break into the role, the ability to run a full local stack—LLM, embeddings, reranker, vector DB—on a single quiet machine is a genuine productivity multiplier. You can test the entire pipeline before a customer ever sees it.

FAQ

Q: Can this run Llama 3.1 405B?

Yes, at Q4_K_M quantization (~230 GB). It’ll be tight—you’ll have minimal room for KV cache—but it runs. At FP16 (810 GB), no.

Q: How does this compare to a Mac Studio with M2 Ultra and 192 GB?

The Mac Studio has significantly higher memory bandwidth (800 GB/s on the M2 Ultra) and will deliver faster tokens per second on large models. But it costs roughly double, runs macOS (which limits some inference tooling), and is not modular or repairable. The Framework board is the tinkerer’s choice.

Q: Can I add a discrete GPU later?

The Desktop case has PCIe slots, so yes—you can add a discrete GPU. But you lose the unified memory advantage for workloads that span both the iGPU and dGPU. The system RAM and VRAM would be separate pools again.

Q: Is ROCm ready for RDNA 3.5?

ROCm 6.1+ supports RDNA 3.5. llama.cpp, vLLM, and PyTorch all have ROCm backends. It’s not as seamless as CUDA, but for inference workloads, it’s production-usable.

Q: What’s the actual price?

Framework hasn’t announced pricing for the 192 GB variant yet. The existing Framework Desktop with 64 GB starts at $1,999. Expect the 192 GB model to land between $2,500 and $3,500 based on the cost of LPDDR5x and AMD’s pricing for the top-bin APU.

Q: Can I use this for training?

You can fine-tune with QLoRA on smaller models (7B-13B), but the memory bandwidth and lack of dedicated tensor cores make full fine-tuning impractical. This is an inference and prototyping machine, not a training rig.