Petals: Running 100B+ LLMs at Home with a BitTorrent-Style Network

What Just Happened: The Petals Protocol

Running a 70-billion-parameter model on a single GPU has been a fantasy for most engineers. You either rent A100s at $3/hour, accept heavily quantized 4-bit versions that lose coherence, or stick to tiny 7B models. The Petals project (https://petals.dev/) changes that equation.

Petals implements a peer-to-peer network where participants collectively serve different layers—called transformer blocks—of a large language model. Your machine holds a few blocks, someone else holds a few others, and the network routes tokens through the chain as if the model were local. The architecture borrows heavily from BitTorrent's philosophy: no central server owns the model; the swarm is the server.

The project currently supports models like Llama 3 70B and Falcon 180B. A user with a single consumer GPU (say, an RTX 3060 with 12GB VRAM) can run inference on these models at interactive speeds—roughly 4-6 tokens per second for Llama 70B, depending on swarm health.

This isn't a theoretical paper. You can install the client library today, connect to the public swarm, and start generating text from a 70B model in under five minutes.

Why This Architecture Matters for Engineers

Three shifts make Petals relevant right now:

1. Inference Beats Training as the Bottleneck

Most organizations aren't training foundation models—they're running inference against them. But inference on large models still demands expensive hardware. Petals distributes inference cost across volunteers, turning a $30,000 hardware problem into a bandwidth problem.

2. The "Private Swarm" Pattern

While the public swarm is compelling, the enterprise angle is stronger. A team of five engineers, each with a mid-range GPU, can form a private Petals swarm and run Llama 70B without sending data to OpenAI or Anthropic. For an FDE working on a customer deployment where data residency matters, this is a concrete architectural option—not just a cool demo. If you're thinking through how FDEs embed with customers and need on-prem LLM access, the patterns in How Palantir-Style FDEs Embed with Customers map cleanly onto this kind of infrastructure.

3. Latency-Hiding Through Pipelining

Petals doesn't wait for all blocks to finish before starting the next token. It uses a pipelined inference scheme: as soon as block 0 finishes processing token t, it ships the hidden state to block 1 and immediately starts on token t+1. This is the same principle that makes GPUs fast—hide latency with parallelism.

Under the Hood: Blocks, DHT, and Fault Tolerance

Large transformer models are stacks of identical blocks. Llama 70B has 80 transformer layers (blocks). Each block is a self-contained unit: it takes hidden states in, applies attention and feed-forward transformations, and outputs new hidden states.

Petals exploits this modularity. The model is sharded by block, not by tensor. Each server in the swarm advertises which blocks it holds via a distributed hash table (DHT), similar to how BitTorrent trackers announce which peers have which file chunks.

The Life of a Single Token

1. Client embeds the input token locally (the embedding layer is tiny and runs on the client).
2. Client queries the DHT to find servers holding blocks 0 through N.
3. Client connects to the server holding block 0, sends the embedding, receives the output hidden state.
4. The chain continues: the client routes that hidden state to block 1's server, then block 2's, and so on.
5. The final server returns logits, and the client runs the lightweight LM head locally to sample the next token.

Fault Tolerance Without Checkpointing

Servers can disappear mid-inference—a peer turns off their machine, a network blip occurs. Petals handles this by maintaining multiple replicas of each block across different peers. If a server times out, the client re-routes to the next available replica for that block. The hidden states for in-flight tokens are discarded, and those tokens are re-processed. It's wasteful but correct.

More importantly, Petals uses chain replication with backup servers. When you serve a block, you can designate a "backup" peer that receives a copy of every hidden state. If you fail, the backup has the exact intermediate state and can resume without recomputation. This is optional but recommended for production-like reliability.

Quantization Strategy

By default, servers store blocks in 8-bit (int8) precision. The forward pass runs in the original precision (typically bfloat16), but the weights are quantized for storage and transfer. This keeps VRAM usage manageable—a single Llama 70B block in int8 fits in roughly 1.2GB, meaning a 12GB GPU can serve 8-10 blocks comfortably.

How to Join the Swarm Today

Running Inference (Client Only)

pip install petals

Then, in Python:

from petals import AutoDistributedModelForCausalLM
from transformers import AutoTokenizer

model_name = "meta-llama/Meta-Llama-3-70B"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoDistributedModelForCausalLM.from_pretrained(
    model_name,
    initial_peers=["/dns/bootstrap.petals.dev/tcp/31337/p2p/QmPeerID"]
)

inputs = tokenizer("Explain backpropagation to a junior engineer:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

The initial_peers argument points to bootstrap nodes that introduce your client to the DHT. Once connected, your client discovers block servers automatically.

Serving Blocks (Contributing to the Swarm)

If you have a GPU with at least 8GB VRAM, you can serve blocks:

python -m petals.cli.run_server meta-llama/Meta-Llama-3-70B \
  --num_blocks 8 \
  --device cuda:0 \
  --attn_cache_tokens 4096

This command downloads 8 blocks of Llama 70B and starts serving them to the swarm. The --attn_cache_tokens flag controls how many tokens of KV cache to retain per sequence—higher values consume more VRAM but support longer generations for clients.

Private Swarm Setup

For a private deployment, skip the public DHT:

# On the coordinator machine:
python -m petals.cli.run_dht --host_maddrs /ip4/0.0.0.0/tcp/31337

# On each server, point to your private DHT:
python -m petals.cli.run_server meta-llama/Meta-Llama-3-70B \
  --initial_peers /ip4/COORDINATOR_IP/tcp/31337/p2p/PEER_ID \
  --num_blocks 16

Clients connect to the same private DHT and never touch the public swarm. This is the setup you'd use for Build a Discord Community FAQ Bot Backed by Your Docs Using Supabase and OpenRouter—swap OpenRouter for your private Petals endpoint and keep everything in-house.

The Balanced Take: Speed, Privacy, and Tradeoffs

What Works Today

• Llama 3 70B at 4-6 tokens/second on a single consumer GPU as a client. That's slow by API standards (OpenAI delivers 30+ t/s) but fast enough for batch processing, summarization, or non-real-time chat.
• Falcon 180B is functional but slower (1-2 t/s) and requires more swarm participants.
• Fine-tuning support exists via parameter-efficient methods (LoRA adapters). You can fine-tune a model distributed across the swarm by only training adapter weights locally.

The Real Constraints

Bandwidth is the bottleneck, not compute. Each token forward pass ships hidden states between every block. For Llama 70B with 80 blocks and a hidden size of 8192, that's roughly 80 × 8192 × 2 bytes (bf16) = 1.3MB per token. At 6 tokens/second, you need ~8MB/s of sustained upload from each server you hit. On residential internet with 20Mbps upload, a single server can handle maybe 2-3 concurrent clients before saturating.

Swarm health is unpredictable. The public swarm works because volunteers run servers. If a popular model loses half its block servers overnight, your inference stalls. Private swarms solve this but require coordination.

Attention is the hidden cost. Each server must maintain a KV cache for every active client sequence. A 4096-token sequence for one Llama 70B block consumes roughly 256MB of VRAM just for the cache. Ten concurrent clients per block means 2.5GB of cache overhead—on top of the 1.2GB for weights. This is why the public swarm sometimes rejects new connections when under load.

Privacy Considerations

Hidden states are not plaintext, but they're not encrypted either. A malicious block server could theoretically extract information from intermediate representations—this is an active research area called "gradient inversion" or "feature inversion." For truly sensitive workloads, stick to private swarms on trusted hardware. If you're building something like a Build a Smart Clipboard That Summarizes and Translates Anything You Copy with Ollama, consider whether the data sensitivity warrants a local-only approach (Ollama) versus a distributed one (Petals).

Where This Fits in the LLM Deployment Landscape

Petals occupies a unique niche: zero marginal cost, large models, and the option of full privacy. It doesn't beat dedicated hardware on speed, but it beats everything else on accessibility.

For FDEs, this is a tool worth knowing. When a customer says "we can't send data to OpenAI but we need a 70B model," Petals on a private swarm is a real answer. The operational patterns—distributed ownership, fault tolerance through replication, bandwidth-aware scheduling—are also directly relevant to Scaling Yourself: When and How an FDE Hands Off to Core Engineering. Understanding distributed inference architecture makes you better at scoping what's feasible in the field.

FAQ

Q: Can I train a model from scratch with Petals? A: No. Petals supports inference and parameter-efficient fine-tuning (LoRA), not full pretraining. The communication overhead of distributed training with gradient synchronization is orders of magnitude higher than inference.

Q: What happens if my internet drops mid-generation? A: The generation fails with a timeout error. Petals doesn't checkpoint client state—you'd need to restart the generation from the beginning. For long-running batch jobs, wrap your inference loop in retry logic.

Q: How does Petals compare to llama.cpp's RPC backend? A: llama.cpp's RPC mode also distributes inference across machines, but it's designed for a known, static cluster (your own servers). Petals adds the DHT-based discovery and fault tolerance layer that makes ad-hoc, volunteer-based swarms possible.

Q: Can I serve multiple models from one GPU? A: Yes, but VRAM is the limit. You can run multiple petals.cli.run_server processes, each serving blocks from a different model, as long as the total VRAM usage fits. Expect roughly 1.2GB per Llama 70B block in int8.

Q: Is there a token limit per request? A: It depends on the servers' --attn_cache_tokens setting. If a server allocated 4096 tokens of KV cache and your request hits 4097, the server will reject the continuation. The client then needs to find a server with a larger cache allocation or fall back to recomputation.

Q: Does Petals work with vision-language models? A: Not natively. The block-serving protocol assumes a pure transformer stack. Vision encoders and cross-attention mechanisms don't decompose cleanly into independent blocks. This is an open research problem.