Build a Voice Assistant for Your Terminal with Local Whisper, Ollama, and Piper TTS

What We’re Building

We’re going to build a voice interface that lives entirely in your terminal. You speak into your mic, the system transcribes your words locally, feeds the text to a local LLM, and speaks the generated response back through your speakers—no cloud API keys, no latency spikes, and no usage bills.

Here’s the concrete feature set:

• Push-to-talk activation via a keyboard key (spacebar) so the assistant only listens when you intend it to.
• Local speech-to-text using OpenAI’s open-source Whisper model. Everything runs on your CPU or GPU, no audio leaves your machine.
• LLM reasoning via Ollama running models like Llama 3 or Mistral. Responses are generated offline.
• Text-to-speech using Piper TTS, a fast, high-quality neural TTS engine that runs locally.
• Terminal-first UX with simple status prints and minimal dependencies.

The whole pipeline is built in Python, and we’ll keep every dependency free and open-source. If you’ve ever wanted a fast, private, always-available voice assistant that feels like your own personal CLI tool, this is the blueprint.

Architecture: The Audio-to-Audio Pipeline

Before we write code, let’s map how the pieces fit together. The system is a linear pipeline with four stages, plus a control loop that gates everything on a keypress.

The flow is straightforward: raw audio hits a gate that only opens when you hold a key. That audio gets transcribed to text by Whisper. The text goes to Ollama, which streams back a response. That response is immediately synthesized into speech by Piper and played through your speakers. Each stage is a separate, swappable component.

Prerequisites and Free Tools

Everything we use is free and runs locally. Here’s what you need installed before we start:

Important setup steps:

1. Install Ollama and pull a model. We recommend llama3:8b for a good speed-quality balance:

   ollama pull llama3:8b
   

   Mistral is also excellent if you want a smaller footprint:

   ollama pull mistral
   

2. Download a Piper voice model. Piper requires a voice model file (.onnx) and its JSON config. Grab one from the Piper releases page. For a natural English voice, download en_US-lessac-medium.onnx and en_US-lessac-medium.onnx.json. Place them in a piper_models/ directory inside your project.

3. Install Piper’s binary. On macOS:

   brew install piper
   

   On Linux, download the prebuilt binary from the Piper releases and add it to your PATH.

Project Setup and Dependencies

Create a fresh directory and set up a virtual environment:

mkdir terminal-voice-assistant
cd terminal-voice-assistant
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

Install Python dependencies:

pip install openai-whisper sounddevice numpy pynput

Create the directory structure:

terminal-voice-assistant/
├── piper_models/
│   ├── en_US-lessac-medium.onnx
│   └── en_US-lessac-medium.onnx.json
├── main.py
└── venv/

We’ll write everything in main.py. The script will be about 150 lines, and we’ll build it section by section.

Step 1: Capturing Live Audio from the Microphone

We’ll use sounddevice to capture audio while the user holds the spacebar. pynput listens for the keypress.

import sounddevice as sd
import numpy as np
from pynput import keyboard
import queue
import threading

# Audio settings
SAMPLE_RATE = 16000
CHANNELS = 1
BLOCK_DURATION = 30  # milliseconds per audio block

audio_queue = queue.Queue()
recording = False

def audio_callback(indata, frames, time, status):
    if recording:
        audio_queue.put(indata.copy())

def on_press(key):
    global recording
    if key == keyboard.Key.space and not recording:
        recording = True
        print("\n🎤 Listening... (release spacebar to stop)")

def on_release(key):
    global recording
    if key == keyboard.Key.space and recording:
        recording = False
        print("✅ Recording stopped.")

def start_keyboard_listener():
    listener = keyboard.Listener(on_press=on_press, on_release=on_release)
    listener.start()
    return listener

This sets up a non-blocking keyboard listener that toggles recording. While recording is True, every audio block gets pushed into audio_queue. We use a sample rate of 16kHz because that’s what Whisper expects.

Step 2: Transcribing Speech with Local Whisper

Once the user releases the spacebar, we drain the queue, concatenate all audio blocks into a single NumPy array, and pass it to Whisper.

import whisper

# Load Whisper model once at startup
whisper_model = whisper.load_model("base.en")

def transcribe_audio(audio_data: np.ndarray) -> str:
    # audio_data is a 1D float32 array at 16kHz
    result = whisper_model.transcribe(audio_data, fp16=False, language="en")
    return result["text"].strip()

We load the base.en model, which is small (142MB), fast, and English-only. It runs comfortably on CPU. If you have a GPU with CUDA, Whisper will use it automatically. For better accuracy, you can swap to small.en or medium.en, but base.en is the sweet spot for real-time terminal use.

Step 3: Generating a Response with Ollama

Ollama exposes a simple HTTP API on localhost:11434. We’ll send the transcribed text as a prompt and stream the response back. This is where the assistant’s personality lives.

import requests
import json

OLLAMA_URL = "http://localhost:11434/api/generate"

def generate_response(prompt: str, model: str = "llama3:8b") -> str:
    system_prompt = (
        "You are a helpful, concise terminal voice assistant. "
        "Keep responses under three sentences. Be direct and technical."
    )
    payload = {
        "model": model,
        "prompt": prompt,
        "system": system_prompt,
        "stream": False
    }
    response = requests.post(OLLAMA_URL, json=payload, timeout=30)
    response.raise_for_status()
    return response.json()["response"].strip()

The system prompt keeps the assistant terse—nobody wants a five-paragraph essay spoken back to them in the terminal. We set stream: False for simplicity, but you could stream tokens and feed them to Piper incrementally if you want lower perceived latency.

Step 4: Speaking the Response with Piper TTS

Piper reads text from stdin and writes WAV audio to stdout. We’ll pipe the LLM response through Piper, capture the output, and play it with sounddevice.

import subprocess
import tempfile
import soundfile as sf
import io

PIPER_BINARY = "piper"
PIPER_MODEL = "piper_models/en_US-lessac-medium.onnx"

def speak_text(text: str):
    # Run piper, passing text via stdin
    cmd = [
        PIPER_BINARY,
        "--model", PIPER_MODEL,
        "--output-raw"
    ]
    process = subprocess.Popen(
        cmd,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.DEVNULL
    )
    # Piper expects text + newline
    raw_audio, _ = process.communicate(input=(text + "\n").encode("utf-8"))
    
    # raw_audio is 16-bit PCM at 22050 Hz (Piper default)
    audio_array = np.frombuffer(raw_audio, dtype=np.int16)
    # Convert to float32 for sounddevice
    audio_float = audio_array.astype(np.float32) / 32768.0
    
    sd.play(audio_float, samplerate=22050)
    sd.wait()

We use --output-raw to get raw PCM samples instead of a WAV file, avoiding unnecessary I/O. The audio is 16-bit signed integer PCM at 22050 Hz, which we convert to float32 for sounddevice.play().

Wiring It All Together: The Main Loop

Now we connect the four stages with a clean main loop. The script waits for a spacebar press, records, transcribes, generates, and speaks.

import time

def main():
    print("🚀 Terminal Voice Assistant Started")
    print("Hold SPACEBAR to speak, release to process. Press ESC to quit.\n")
    
    listener = start_keyboard_listener()
    
    # Open an input stream that continuously feeds the callback
    stream = sd.InputStream(
        samplerate=SAMPLE_RATE,
        channels=CHANNELS,
        callback=audio_callback,
        blocksize=int(SAMPLE_RATE * BLOCK_DURATION / 1000)
    )
    stream.start()
    
    try:
        while True:
            # Wait for recording to start
            while not recording:
                time.sleep(0.05)
            
            # Collect audio while recording is True
            audio_chunks = []
            while recording:
                try:
                    chunk = audio_queue.get(timeout=0.1)
                    audio_chunks.append(chunk)
                except queue.Empty:
                    continue
            
            if not audio_chunks:
                print("⚠️ No audio captured.")
                continue
            
            # Concatenate and flatten
            audio_data = np.concatenate(audio_chunks, axis=0).flatten()
            
            # Step 2: Transcribe
            print("📝 Transcribing...")
            prompt = transcribe_audio(audio_data)
            print(f"🗣️ You said: \"{prompt}\"")
            
            if not prompt:
                print("⚠️ Couldn't understand. Try again.")
                continue
            
            # Step 3: Generate
            print("🤖 Generating response...")
            response = generate_response(prompt)
            print(f"💬 Assistant: \"{response}\"")
            
            # Step 4: Speak
            print("🔊 Speaking...")
            speak_text(response)
            print("✅ Done. Ready for next query.\n")
            
    except KeyboardInterrupt:
        print("\n👋 Shutting down.")
    finally:
        stream.stop()
        stream.close()
        listener.stop()

if __name__ == "__main__":
    main()

That’s the entire assistant. The loop is event-driven: it idles until the spacebar goes down, captures audio while it’s held, and runs the full pipeline on release. The ESC key or Ctrl+C exits cleanly.

Running Your Terminal Voice Assistant

Make sure Ollama is running in a separate terminal:

ollama serve

Then launch the assistant:

python main.py

Hold the spacebar, speak a question like “What’s the capital of France?”, release, and wait for the spoken response. The terminal prints status at each stage so you always know what’s happening.

On first run, Whisper downloads the base.en model to ~/.cache/whisper/. Subsequent runs are instant.

Sensible Extensions

Once the core loop works, you can extend it in powerful directions:

• Wake word detection: Replace the spacebar with an always-on wake word detector like Porcupine or OpenWakeWord. This makes the assistant fully hands-free.
• Streaming TTS: Pipe Ollama’s token stream directly into Piper’s stdin as tokens arrive. This cuts perceived latency in half—the assistant starts speaking before the full response is generated.
• Tool use: Add a system prompt that teaches the LLM to output structured commands. Parse those commands to control your terminal: open files, run shell commands, or query local databases. This is the same pattern we used in the Build a SQL Analyst Agent That Answers Questions Over a Postgres Database with LlamaIndex and Groq guide.
• Conversation history: Maintain a rolling buffer of past exchanges and include them in the Ollama prompt. The assistant gains memory across turns.
• Multi-voice support: Download multiple Piper voice models and let the user switch voices with a command.
• Agent delegation: For complex research tasks, you could chain this voice interface to a multi-agent system like the one we built in Build a Multi-Agent Research Assistant That Plans, Searches, and Writes a Brief with Groq and Tavily.

Common Pitfalls and Debugging Tips

“No module named ‘sounddevice’” — You need PortAudio installed at the system level. On macOS: brew install portaudio. On Ubuntu: sudo apt install portaudio19-dev. Then reinstall sounddevice: pip install --force-reinstall sounddevice.

Piper says “model not found” — Double-check the path to your .onnx file. Use an absolute path if relative isn’t resolving: PIPER_MODEL = "/full/path/to/piper_models/en_US-lessac-medium.onnx".

Ollama connection refused — Ensure ollama serve is running. Test with curl http://localhost:11434/api/tags. If you see a connection error, Ollama isn’t running or is on a different port.

Whisper transcription is slow — The base.en model should transcribe a 5-second clip in under 2 seconds on a modern CPU. If it’s slower, try tiny.en. If you have a GPU, install PyTorch with CUDA and Whisper will use it automatically.

Audio sounds choppy or robotic — Piper’s default sample rate is 22050 Hz. Make sure you’re playing back at that exact rate. If you used --output-raw, the raw PCM is 16-bit mono at 22050 Hz. Any mismatch causes distortion.

Keyboard listener blocks on macOS — On macOS, pynput needs Accessibility permissions. Go to System Settings > Privacy & Security > Accessibility and grant permission to your terminal app.

FAQ

Q: Does this work offline? A: Yes, completely. Whisper, Ollama, and Piper all run locally. The only network call is Ollama’s initial model pull, which you do once. After that, airplane mode works fine.

Q: Can I use a different LLM? A: Absolutely. Swap llama3:8b for any model in your Ollama library: mistral, phi3, gemma:7b, etc. Smaller models give faster responses; larger models give better reasoning. Adjust the system prompt to match the model’s strengths.

Q: How do I make it respond faster? A: Three levers: (1) Use a smaller Whisper model like tiny.en. (2) Use a smaller Ollama model like phi3 or llama3.2:1b. (3) Implement streaming TTS so playback starts before the full response is generated.

Q: Can I run this on a Raspberry Pi? A: Yes, with caveats. Use tiny.en for Whisper and a very small Ollama model like tinyllama. Piper runs well on ARM. Expect 5-10 seconds of latency per query on a Pi 4. It’s usable but not snappy.

Q: The assistant doesn’t stop talking—how do I interrupt it? A: The current implementation blocks on sd.wait(). To add interruptibility, run speak_text in a separate thread and kill it on a new keypress. This is a natural next step if you’re building a daily driver.

Q: Where can I learn more about building AI-powered CLI tools? A: If you enjoy wiring together local models into practical tools, you’ll find the same hands-on, zero-fluff approach in our guide on Build an Email Cold-Outreach Personalizer That Reads a CSV of Prospects Using Groq's Free Tier. For deeper dives into the engineering mindset behind these builds, check out What a Forward Deployed Engineer Actually Does in a Week at an AI Startup.