Mesh LLM: Peer-to-Peer Distributed Inference for Local-First AI Clusters

What Actually Happened: iroh’s P2P Inference Backbone

Number Zero released mesh-llm, an open-source project that distributes large language model inference across a peer-to-peer network. Instead of routing every token through a centralized API endpoint, mesh-llm discovers other nodes on a local network or over the internet, shards the model layers, and streams activations directly between peers. The transport layer is built entirely on iroh, a modern networking library that provides QUIC-based connectivity, content-addressed data transfer, and a decentralized identity system.

The demo is deliberately small-scale: two or three laptops splitting a Llama-family model. One machine holds the embedding layer and early transformer blocks, another handles middle layers, and a third completes the final projection. The output is a stream of generated tokens stitched together so the user sees a single coherent response. No Kubernetes, no load balancer, no API key. Just a peer ID and a shared model.

What makes this different from prior distributed inference attempts is the local-first philosophy. iroh treats network connections as an implementation detail, not a permission boundary. Peers authenticate via Ed25519 keys, exchange data over encrypted QUIC streams, and can even sync model weights through iroh’s blob-sync protocol. The result is a system that feels like running a local model, but can borrow compute from trusted machines on the fly.

Source: iroh.computer/blog/mesh-llm

Why It Matters for Forward-Deployed Engineers

FDEs live in the gap between a customer’s infrastructure and the theoretical promise of AI. You’re the person who has to make an LLM feature work inside a hospital network that blocks outbound API calls, or on a factory floor with intermittent connectivity. Mesh LLM speaks directly to that reality.

Offline-first inference. The most obvious win is eliminating the cloud dependency. If your customer has five workstations on a closed VLAN, mesh-llm lets those machines pool their GPUs and run a model that none of them could load individually. No internet required. This isn’t just a latency optimization—it’s a compliance enabler. Many regulated environments (defense, healthcare, finance) mandate that data never leaves the premises. With mesh-llm, the entire inference pipeline stays inside the air-gapped network.

Dynamic resource pooling. In a typical enterprise deployment, you over-provision a GPU server to handle peak load, and it sits idle 90% of the time. Mesh LLM turns that model inside out. When a developer’s laptop is compiling code, its GPU is available. When a designer steps away for lunch, their workstation can serve transformer layers. The cluster’s capacity is the sum of whatever machines happen to be online. For an FDE prototyping a customer solution, this means you can demonstrate a 13B-parameter model on hardware the customer already owns, without waiting for procurement to approve a cloud budget.

A new deployment topology. If you’ve ever wrestled with deploying an LLM feature at an enterprise customer, you know that infrastructure friction kills more pilots than model quality ever does. Mesh LLM sidesteps the traditional server-client architecture entirely. There’s no “deployment” in the traditional sense—just a binary or Python script that discovers peers and starts contributing. This aligns with the messy-customer-problem-to-shipped-prototype workflow that FDEs excel at: find the available compute, wire it together, and show value fast.

Architecture Deep-Dive: The iroh Data Flow

Understanding mesh-llm requires understanding iroh’s three core primitives: nodes, blobs, and documents.

Nodes and identity. Every peer in an iroh network has a cryptographic identity derived from an Ed25519 keypair. This identity is stable across network changes—a laptop moving from office Wi-Fi to home VPN keeps the same peer ID. Mesh LLM uses these identities to build a deterministic routing table: each peer knows exactly which layers it owns and which peer holds the next chunk of the model.

Blob sync for model weights. Before inference starts, all peers need the same model. iroh’s blob protocol handles this with content-addressed chunking. The model is split into verified chunks, and peers fetch only the chunks relevant to their assigned layers. If Peer A already has the embedding weights, Peer B can pull just the middle-layer tensors from Peer A over an encrypted QUIC connection. This is bandwidth-efficient and resumable—if a peer disconnects mid-sync, it picks up where it left off.

Streaming activations. During inference, the forward pass flows sequentially through the pipeline. Peer A embeds the input tokens and streams the hidden states to Peer B. Peer B runs its transformer blocks and streams the result to Peer C. Peer C computes logits, samples a token, and feeds it back to Peer A for the next autoregressive step. The entire pipeline is synchronous per-token, meaning latency is bounded by the slowest peer. But because each step is a small tensor (hidden dimension × sequence length), the actual data transfer is on the order of kilobytes per token.

Failure handling. If a peer drops, the pipeline stalls. The current implementation doesn’t do automatic rebalancing—it’s a research-grade demo, not a production system. But the iroh primitives make rebalancing feasible: since model weights are content-addressed, a new peer can join, fetch the required chunks, and advertise its availability through the document sync layer.

Hands-On: Running Mesh LLM on Your Local Cluster

The repo is at github.com/n0-computer/mesh-llm (MIT licensed). Here’s what you need to know to get it running on two or three machines.

Prerequisites. Each machine needs a GPU with enough VRAM for its assigned model slice. For a 7B model split two ways, 8GB GPUs work; for three-way, 6GB cards are sufficient. You’ll need Rust (the project is written in pure Rust on top of iroh and Candle, HuggingFace’s Rust-native ML framework).

Step 1: Clone and build.

git clone https://github.com/n0-computer/mesh-llm
cd mesh-llm
cargo build --release

Step 2: Launch the first peer (the coordinator). This peer will load the embedding layer and the first few transformer blocks. The --layers flag specifies which model layers this peer owns.

./target/release/mesh-llm serve \
  --model llama-7b \
  --layers 0..16 \
  --bind 0.0.0.0:9090

The terminal prints a peer ID and a ticket. The ticket is a base64-encoded string that encodes the peer ID, addresses, and a shared secret for initial discovery. Share this ticket with your other machines.

Step 3: Join additional peers. On each additional machine, run the same binary with a different layer range and the ticket from the first peer.

./target/release/mesh-llm serve \
  --model llama-7b \
  --layers 16..32 \
  --ticket "bl3u...base64ticket...abc="

Step 4: Send a prompt. Any peer can initiate inference. Use the chat subcommand:

./target/release/mesh-llm chat \
  --ticket "bl3u...base64ticket...abc=" \
  --prompt "Explain BGP route reflection in one paragraph."

The prompt is tokenized locally, forwarded through the peer pipeline, and the generated tokens stream back to your terminal.

What you’ll observe. The first token takes noticeably longer than subsequent ones. This is the cold-start overhead of streaming activations through multiple machines. Once the pipeline is warm, token generation speed depends on the slowest peer’s GPU throughput plus network latency. On a wired gigabit LAN with two RTX 3060s splitting a 7B model, expect roughly 60-70% of single-machine throughput. The gap is almost entirely network serialization overhead—Candle tensors are serialized to a compact binary format, but you’re still moving hidden states across a socket for every token.

Troubleshooting. If peers fail to connect, check that UDP port 9090 is open between machines (QUIC runs over UDP). If model sync hangs, verify that all peers have enough disk space for the model weights—iroh caches blobs locally. For detailed logs, set RUST_LOG=debug.

A Balanced Take: Strengths, Limits, and the Road Ahead

Mesh LLM is not production-ready. It’s a proof of concept that demonstrates a genuinely new deployment model. Here’s an honest assessment.

Strengths.

• Zero-infrastructure inference. No Docker, no Kubernetes, no API gateway. Two machines on the same subnet discover each other and start collaborating. This is the closest thing to “it just works” that distributed inference has achieved.
• Security model. iroh’s cryptographic peer identities mean you’re not accidentally connecting to a malicious node. The ticket-based discovery is a one-time authorization mechanism—once peers have exchanged keys, subsequent connections are mutually authenticated.
• Local-first architecture. The entire system is designed around the assumption that the network is unreliable and peers come and go. This is philosophically aligned with how real-world edge deployments operate.
• Rust performance. Candle and iroh are both written in Rust, meaning no Python GIL contention, no serialization overhead from protobuf or JSON, and direct memory-mapped tensor access.

Limitations.

• Synchronous pipeline bottleneck. Every token must traverse the full peer chain sequentially. If Peer B is 20% slower than Peer A, the entire system runs at Peer B’s speed. There’s no pipelining across tokens—you can’t start computing token N+1 until token N is fully generated.
• Fixed sharding. The layer assignment is static and manual. If a peer with layers 8-16 disconnects, the pipeline halts. Dynamic rebalancing would require redistributing weights and re-establishing the activation flow, which is a hard distributed systems problem.
• No KV-cache sharing. In standard autoregressive inference, the key-value cache from previous tokens is reused to avoid recomputation. In mesh-llm, each peer’s KV cache stays local to that peer, but the pipeline structure means the cache for early layers must be available when the next token loops back. The current implementation recomputes some attention states, reducing throughput.
• Limited to Llama-family models. The layer splitting logic is written for the standard transformer architecture (embedding → N × transformer block → LM head). Models with non-standard architectures (mixture-of-experts, Mamba) would require significant rework.

Where this goes next. The iroh team has hinted at several directions: speculative decoding across peers (where a smaller draft model runs locally and a larger verifier model runs remotely), automatic layer partitioning based on peer GPU profiles, and integration with iroh’s document sync for live model weight updates. For FDEs, the most exciting near-term possibility is a lightweight inference runtime that you can drop onto any machine in a customer environment and have it immediately contribute to a shared model pool—no configuration, no central server, just a peer ID and available VRAM.

If you’re building agentic systems that need to run locally, this approach pairs naturally with architectures like the multi-agent research assistant pattern—imagine a coordinator agent running on a laptop that offloads heavy inference to peer GPUs on the same network. Or consider a PR review bot that runs entirely on-premises, using idle developer workstations to review code without ever touching a cloud API.

FAQ

Does mesh-llm work over the internet, or only on a LAN?

It works over any IP network where peers can establish QUIC connections. On a LAN, discovery is near-instant via UDP broadcast. Over the internet, you’ll need to ensure UDP port connectivity between peers (or use a relay). iroh includes relay server support for NAT traversal, similar to Tailscale’s DERP relays.

Can I use this with a GPU server and a CPU-only laptop?

Yes, but the CPU peer will be the bottleneck. You can assign the embedding layer and LM head (which are relatively lightweight) to the CPU machine, and put the heavy transformer blocks on the GPU server. The pipeline will still be limited by the CPU’s activation processing speed.

How does this compare to llama.cpp’s RPC backend?

llama.cpp’s RPC mode also does distributed inference, but it follows a client-server model: one machine runs an RPC server and offloads specific layers to remote GPUs. Mesh LLM is fully peer-to-peer with no designated server. The iroh approach also provides content-addressed weight sync and cryptographic peer identities, which llama.cpp’s RPC doesn’t address.

What happens if two peers have different GPU architectures?

Candle handles cross-device tensor transfers, but all peers must use the same model format and dtype. If one peer has an NVIDIA GPU (CUDA) and another has an Apple Silicon GPU (Metal), the tensors are transferred in a device-agnostic format and each peer re-allocates on its own device. This works but adds serialization overhead.

Is there a Python API, or is it Rust-only?

The core is Rust, but iroh has Python bindings (iroh-python). The mesh-llm inference logic (layer splitting, tensor streaming) would need to be ported. For quick experimentation, the Rust binary is the path of least resistance.

Can I use this in production today?

No. The synchronous pipeline, static sharding, and lack of KV-cache optimization make it unsuitable for production workloads. It’s a research prototype that demonstrates the architecture. Treat it as a glimpse of what local-first distributed inference will look like in 12-18 months, not something to deploy to customers.