LLM on an $8 ESP32: How a 29M-Param Model Runs on a Microcontroller

What Actually Happened: The Specs That Matter

A developer going by the handle slvDev shipped a working implementation of a 28.9-million-parameter large language model running entirely on an ESP32-S3 microcontroller. The dev board costs roughly $8. No cloud, no GPU, no network call. The model is a distilled, heavily quantized version of Microsoft's Phi-1.5, and it generates coherent text at about 8–12 tokens per second.

Let's put that in perspective: the ESP32-S3 has 512 KB of SRAM, 8 MB of external PSRAM, and a dual-core Xtensa LX7 processor running at 240 MHz. It's a chip designed for IoT sensors, not transformer inference. A year ago, the idea of running any transformer on this class of hardware was laughable. Now you can flash firmware and have a locally running chatbot that fits in your palm.

The repo—esp32-ai on GitHub—includes the full firmware, a pre-converted GGUF model, and a platform.io project that compiles cleanly. The model responds to prompts like "What is the capital of France?" with reasonable, if terse, answers. It's not ChatGPT. It's something more interesting: a proof point that the lower bound for useful on-device inference has dropped through the floor.

The Engineering Stack: How It Works Under the Hood

This isn't magic. It's a carefully stacked set of optimizations that each shave off just enough memory and compute to make the impossible possible. Here's the architecture:

Distillation: Shrinking the Giant

The starting point is Microsoft's Phi-1.5, a 1.1-billion-parameter model trained primarily on high-quality textbook data. Through knowledge distillation—training a much smaller student model to mimic the output distribution of the larger teacher—the model was compressed to 28.9 million parameters. That's a 38x reduction. The student loses breadth and nuance, but retains basic syntactic structure and factual recall for common knowledge.

Quantization: From FP16 to 4 Bits

The big win comes from quantization. The original model weights are stored as 16-bit floating-point numbers. At 29M parameters, that's roughly 58 MB just for weights—already exceeding the ESP32's PSRAM. Enter Q4_K_M quantization via the GGML/GGUF ecosystem. This scheme stores most weights at 4 bits, with some higher-precision outliers kept at 6 bits to preserve important attention heads. The result: the full model fits in about 16 MB, comfortably within the 8 MB PSRAM after streaming decompression and careful memory mapping.

The Inference Engine: llama.cpp on Bare Metal

The runtime is a port of llama.cpp—the same C++ inference engine that powers local LLMs on MacBooks and Raspberry Pis. The ESP32 port strips out everything non-essential: no file I/O, no threading beyond the two cores, no dynamic memory allocation during inference. The model weights are memory-mapped directly from PSRAM. The KV cache, which stores attention key-value pairs for all previous tokens, lives in the precious 512 KB of SRAM.

This is where the token-per-second ceiling comes from. With only 512 KB of fast SRAM, the KV cache can only hold context for about 256–512 tokens before it overflows. The implementation uses a sliding window attention mechanism: when the cache fills, the oldest tokens get evicted. This means the model has limited long-range coherence, but for short-form Q&A, it's sufficient.

Token Generation Loop

The inference loop is brutally simple:

1. Tokenize the input prompt using a minimal SentencePiece tokenizer baked into the firmware.
2. For each new token, load the corresponding weight row from PSRAM into the CPU's register file.
3. Compute the attention scores against the KV cache in SRAM.
4. Run the feed-forward layers, again streaming weights from PSRAM.
5. Sample from the output logits using top-k sampling (k=40) to avoid low-probability gibberish.
6. Append the new token's key-value pair to the KV cache, evicting oldest if necessary.
7. Repeat until an end-of-sequence token or max length.

No speculative decoding, no flash attention, no batching. Just a tight C++ loop running on two 240 MHz Tensilica cores. At 8–12 tokens per second, a 50-token response takes about 4–6 seconds. Slow by server standards, but fast enough for an interactive device.

Why This Matters for Engineers and FDEs

For embedded engineers, this is a clear signal: on-device language intelligence is no longer reserved for $100+ SBCs with heatsinks. The ESP32-S3 is a chip you can put in a light switch, a doorbell, or a $15 gadget. Running a model locally means zero latency for network round-trips, no cloud costs, and complete data privacy.

For Forward Deployed Engineers (FDEs), the implications are more strategic. FDEs are the engineers who sit between a company's product and the customer's environment, bending the product to solve real problems. This capability unlocks a class of edge deployments that were previously off-limits:

• Offline-first voice interfaces: A factory floor with no WiFi can still have a voice-controlled troubleshooting assistant that runs entirely on-device.
• Privacy-sensitive environments: Hospitals, law firms, and government facilities often prohibit data leaving the premises. A local LLM on cheap hardware sidesteps the compliance nightmare.
• BOM-sensitive consumer products: Adding an $8 chip and a speaker is a viable BOM line item. Adding a $35 Raspberry Pi and a cloud dependency is not.

If you're building an FDE portfolio to demonstrate deployment velocity, an edge inference project like this is gold. It shows you understand the full stack—model conversion, quantization, embedded C++ toolchains, and real-world constraints. Check out our guide on the FDE portfolio: what to build to demonstrate deployment velocity and get hired for more project ideas that signal the right skills.

How to Try It Yourself Today

You need three things: an ESP32-S3 dev board with at least 8 MB of PSRAM (the ESP32-S3-DevKitC-1 or any compatible board works), a USB-C cable, and PlatformIO installed.

Step 1: Clone and Compile

git clone https://github.com/slvDev/esp32-ai.git
cd esp32-ai

Open the project in PlatformIO (VS Code extension recommended). The platformio.ini is pre-configured for the ESP32-S3 with PSRAM enabled. Hit build. The first compile will pull the Arduino-ESP32 core, llama.cpp port, and all dependencies.

Step 2: Flash the Model

The pre-quantized GGUF model is too large to store in the repo itself. The project includes a download script that fetches it from a release. Run:

pio run --target uploadfs

This uploads the model binary to the ESP32's flash filesystem (SPIFFS). At boot, the firmware copies it into PSRAM for inference.

Step 3: Connect and Prompt

Connect to the ESP32's serial monitor at 115200 baud:

pio device monitor

You'll see a prompt. Type a question and hit enter. The model processes and streams tokens back over serial. Expect 8–12 tokens per second. The demo firmware includes a simple read-eval-print loop (REPL) that handles tokenization and sampling.

Step 4: Customize the Model

Want to swap in a different GGUF model? Any llama.cpp-compatible model that fits in ~6 MB after quantization should work. Convert it using the standard convert.py and quantize tools from the llama.cpp project, then replace the file in the data/ directory before running uploadfs. Domain-specific models—say, one fine-tuned on your company's internal documentation—would make this a powerful offline knowledge assistant. Speaking of which, if you're interested in building knowledge assistants that work with your own data, our guide on building a Notion knowledge assistant that answers questions from your workspace walks through a complementary approach using Supabase and LlamaIndex.

The Balanced Take: Limits and Reality

Let's be clear about what this is and isn't. This is an engineering achievement, not a product. The 29M-parameter model has severe limitations:

• Factual reliability is low. It knows Paris is the capital of France, but ask it who won the 2022 World Cup and it may hallucinate. There's no retrieval-augmented generation (RAG), no grounding, no safety fine-tuning beyond what was in the base Phi-1.5 distillation.
• Context window is tiny. With only 512 KB of SRAM for the KV cache, the effective context is 256–512 tokens. It can't summarize a document or maintain a multi-turn conversation with history.
• No tool use, no function calling. This is a pure text-in, text-out model. It can't control GPIO pins, read sensors, or trigger actions. That logic would need to be built in a separate control loop that parses the model's output.
• Inference speed is at the edge of usable. 8–12 tokens per second is fine for a single-sentence answer. It's not okay for anything requiring real-time interaction or long-form generation.
• Power consumption is non-trivial for battery operation. The ESP32-S3 at full tilt draws ~200–300 mA. A CR2032 coin cell won't cut it. You need a LiPo or wall power for anything beyond a demo.

That said, the trajectory is unmistakable. Two years ago, the smallest useful LLM was 1B parameters and needed a phone-class SoC. Now we're at 29M on a microcontroller. The techniques that got us here—distillation, aggressive quantization, streaming weight loading, sliding window attention—are all still improving. 10M-parameter models with 1K context windows at 20+ tokens per second are likely within 12–18 months.

For FDEs, the play isn't to ship this exact setup to a customer. It's to recognize that the building blocks are now cheap enough to prototype edge AI features that were science fiction last quarter. The FDE who can say "we can run a language model on the existing BOM, no cloud dependency" has a superpower in regulated or cost-sensitive deployments. If you're preparing for that kind of conversation, our breakdown of the FDE interview loop: how to prepare for deployments, debugging, and demos covers exactly how to present technical tradeoffs like these under pressure.

And if you're already thinking about how to integrate this with real workflows, consider the pattern: edge device collects data, local LLM processes or summarizes, and a lightweight agent handles the cloud interaction when connectivity is available. We've covered a similar architecture in our guide to building an on-call incident summarizer from logs with Groq and Playwright—the same principle of local-first, cloud-augmented applies.

FAQ

Q: Can I run this on a regular ESP32 (non-S3)? A: No. The original ESP32 lacks the vector extensions and PSRAM bandwidth needed. You need the S3 variant with at least 8 MB of PSRAM.

Q: What's the largest model I can realistically run? A: With 8 MB PSRAM and Q4 quantization, the practical ceiling is around 35–40M parameters. Beyond that, you'd need an ESP32-P4 or an external PSRAM chip, which adds BOM cost and complexity.

Q: Can it do real-time speech-to-text or text-to-speech? A: Not in the current firmware. The token generation is purely text-based. You'd need separate models for ASR and TTS, and running all three on one ESP32-S3 simultaneously isn't feasible with current memory constraints.

Q: Is the model fine-tunable? A: You can fine-tune the 29M student model on a GPU using standard tools (LoRA works well at this scale), then quantize and convert to GGUF for deployment. The repo doesn't include training scripts, but the model architecture is standard transformer, so any PyTorch fine-tuning pipeline works.

Q: What's the latency for the first token? A: About 1–2 seconds. Most of that is the initial prompt processing, which requires loading all prompt token weights from PSRAM before the first new token can be generated.

Q: How does this compare to running llama.cpp on a Raspberry Pi Zero 2 W? A: The Pi Zero 2 W has 512 MB of RAM and a 1 GHz ARM core. It can run 100M+ parameter models at 2–4 tokens per second with more context. The ESP32-S3 wins on cost, power, and physical footprint; the Pi Zero 2 W wins on model capacity and flexibility.