Build a Free, Offline Voice Assistant for Your Terminal with Whisper, Ollama, and Piper

What We're Building

We'll construct a terminal-resident voice assistant that runs entirely on your machine, using no paid APIs. You speak into your mic, it transcribes with Whisper, reasons with a local LLM via Ollama, and speaks back through Piper TTS. The whole pipeline is orchestrated in a single Python event loop.

Feature list:

• Hotword-free, push-to-talk activation (press Enter, speak, release)
• Streaming mic capture via sounddevice
• Speech-to-text with OpenAI Whisper (base or small model)
• LLM command processing with Ollama (Codestral or Llama 3)
• Text-to-speech output with Piper TTS
• Completely offline-capable after initial model pulls
• Modular Python classes you can swap or extend

This isn't a toy. It's a foundation you can wire into your dotfiles, pipe to shell commands, or use as a pair-programming rubber duck that actually talks back. If you've ever wondered how to chain local models into a coherent voice pipeline without touching a cloud GPU, this guide is your blueprint.

Architecture Overview

Before we type a single line, let's nail down the data flow. Understanding the event loop and buffer handoffs will save you from debugging silent failures later.

The architecture is a linear pipeline with three heavy-lift components. sounddevice captures raw PCM audio into a NumPy buffer. We write that buffer to a temporary WAV file because Whisper expects file input. Whisper returns text, which we inject into an Ollama chat completion call. The LLM's response string feeds directly into Piper's piper CLI, which synthesizes a WAV that we play with sounddevice.

Key design decisions:

• File I/O for model boundaries keeps each component testable in isolation. You can swap Whisper for faster-whisper or Piper for espeak without touching the loop.
• Synchronous execution with a simple while True loop. For a single-user terminal assistant, async adds complexity without latency benefit—the LLM is the bottleneck, not I/O.
• Push-to-talk avoids the CPU drain of continuous hotword detection. You control when the assistant listens.

Prerequisites and Free Tools

Every tool here is free and open-source. No credit card required.

Whisper model sizing: The base model (~142 MB) runs on CPU and transcribes in near-real-time on any machine from the last 5 years. If you have a GPU, pull small (~461 MB) for better accuracy. Avoid tiny—its word error rate on technical vocabulary will frustrate you.

Ollama model choice: Pull codestral if you want the assistant to help with code and shell commands. Pull llama3 for general-purpose conversation. After installing Ollama, run:

ollama pull codestral  # or llama3

Piper voices: Piper ships with a default English voice (en_US-lessac-medium). To list available voices:

piper --list-models

Download a voice once: piper --model en_US-lessac-medium --download. The voice file lands in ~/.local/share/piper-tts/.

Step-by-Step Implementation

We'll build a single Python script, voice_assistant.py. It's ~120 lines, fully commented, and you can run it immediately after installing dependencies.

1. Imports and Configuration

import sys
import wave
import tempfile
import subprocess
import numpy as np
import sounddevice as sd
import whisper
import ollama

# ---- CONFIG ----
SAMPLE_RATE = 16000  # Whisper expects 16kHz
CHANNELS = 1
DTYPE = 'int16'
SILENCE_THRESHOLD = 0.01  # Amplitude threshold for auto-stop
MAX_RECORD_SECONDS = 15
WHISPER_MODEL = "base"  # or "small"
OLLAMA_MODEL = "codestral"  # or "llama3"
PIPER_VOICE = "en_US-lessac-medium"

2. Audio Recording with Auto-Stop

We use sounddevice.InputStream to capture audio into a ring buffer. The record_until_silence function stops when the RMS amplitude drops below threshold for 1.5 seconds, or when the user presses Enter again.

def record_until_silence():
    """Record from default mic. Returns raw bytes of WAV data."""
    print("🎤 Listening... (press Enter to stop manually)")
    
    audio_buffer = []
    silence_counter = 0
    silence_frames = int(1.5 * SAMPLE_RATE / 1024)  # 1.5s of 1024-sample chunks
    
    def callback(indata, frames, time, status):
        nonlocal silence_counter
        audio_buffer.append(indata.copy())
        rms = np.sqrt(np.mean(indata**2))
        if rms < SILENCE_THRESHOLD:
            silence_counter += 1
        else:
            silence_counter = 0
        if silence_counter > silence_frames:
            raise sd.CallbackStop
    
    with sd.InputStream(samplerate=SAMPLE_RATE, channels=CHANNELS,
                        dtype=DTYPE, callback=callback):
        try:
            sd.sleep(MAX_RECORD_SECONDS * 1000)
        except sd.CallbackStop:
            pass
    
    if not audio_buffer:
        return None
    
    audio_data = np.concatenate(audio_buffer, axis=0)
    return audio_data.tobytes()

The CallbackStop exception is the cleanest way to exit the stream from inside the callback. The alternative—polling a flag from the main thread—introduces race conditions.

3. Speech-to-Text with Whisper

Whisper reads from a file path. We dump the raw bytes into a temporary WAV with proper headers.

def transcribe_audio(audio_bytes, model):
    """Write audio to temp WAV, transcribe with Whisper."""
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
        with wave.open(f, 'wb') as wf:
            wf.setnchannels(CHANNELS)
            wf.setsampwidth(2)  # 16-bit = 2 bytes
            wf.setframerate(SAMPLE_RATE)
            wf.writeframes(audio_bytes)
        temp_path = f.name
    
    result = model.transcribe(temp_path, fp16=False, language="en")
    import os; os.unlink(temp_path)
    return result["text"].strip()

Set fp16=False unless you have a CUDA-capable GPU. On Apple Silicon, Whisper uses CoreML automatically if available.

4. LLM Processing with Ollama

We use Ollama's Python library. It communicates with the local Ollama server over HTTP on localhost:11434.

def process_with_llm(text):
    """Send transcribed text to Ollama, return response."""
    system_prompt = (
        "You are a concise terminal assistant. Answer in 1-3 sentences. "
        "If the user asks for a shell command, provide only the command "
        "with a brief explanation. Never use markdown formatting."
    )
    response = ollama.chat(
        model=OLLAMA_MODEL,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": text}
        ]
    )
    return response['message']['content']

That system prompt is load-bearing. Without it, Llama 3 will happily write you a three-paragraph essay when you ask "what's the weather." Tune it to your taste—add personality, domain knowledge, or shell-execution formatting.

5. Text-to-Speech with Piper

Piper runs as a subprocess. We pipe text in via stdin and capture WAV output on stdout, then play it with sounddevice.

def speak_text(text):
    """Synthesize text with Piper and play through speakers."""
    cmd = [
        "piper",
        "--model", PIPER_VOICE,
        "--output-raw"
    ]
    proc = subprocess.Popen(cmd, stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
    stdout, _ = proc.communicate(input=text.encode('utf-8'))
    
    # Piper --output-raw sends 16-bit PCM at 22050 Hz
    audio_np = np.frombuffer(stdout, dtype=np.int16)
    # Resample to a common playback rate if needed; most hardware handles 22050
    sd.play(audio_np, samplerate=22050)
    sd.wait()

--output-raw skips WAV header generation, giving us raw PCM bytes. Piper's default sample rate is 22050 Hz. If your hardware balks, add a scipy.signal.resample step, but in practice modern DACs handle it fine.

6. Main Event Loop

def main():
    print("Loading Whisper model...")
    whisper_model = whisper.load_model(WHISPER_MODEL)
    print(f"Whisper '{WHISPER_MODEL}' loaded.")
    print(f"Using Ollama model: {OLLAMA_MODEL}")
    print("\n=== Terminal Voice Assistant Ready ===")
    print("Press Enter to start speaking, then press Enter again or stay silent to stop.\n")
    
    while True:
        try:
            input("Press Enter to speak...")
            audio_bytes = record_until_silence()
            if audio_bytes is None or len(audio_bytes) < SAMPLE_RATE * 0.5:
                print("No speech detected.\n")
                continue
            
            print("Transcribing...")
            text = transcribe_audio(audio_bytes, whisper_model)
            if not text:
                print("Couldn't transcribe.\n")
                continue
            print(f"You said: {text}")
            
            print("Thinking...")
            response = process_with_llm(text)
            print(f"Assistant: {response}")
            
            print("Speaking...")
            speak_text(response)
            print("Done.\n")
            
        except KeyboardInterrupt:
            print("\nGoodbye.")
            sys.exit(0)
        except Exception as e:
            print(f"Error: {e}\n")

if __name__ == "__main__":
    main()

Running the Assistant

1. Start Ollama in a separate terminal (if not running as a service):

   ollama serve
   

2. Run the assistant:

   python voice_assistant.py
   

3. Workflow: Press Enter → speak your query ("list all Python processes") → go silent or press Enter again → watch transcription appear → hear the response.

First-run latency note: Whisper loads the model on startup (3-10 seconds depending on hardware). Ollama's first inference also triggers model loading if it's not cached in RAM. Subsequent queries are fast.

Troubleshooting audio devices: If sounddevice can't find your mic, list devices:

import sounddevice as sd
print(sd.query_devices())

Then set sd.default.device = [input_device_id, output_device_id] before the stream.

Sensible Extensions

Once the loop runs, the real fun starts. Here are three high-impact extensions that turn this from a demo into a daily driver:

1. Shell command execution. Parse the LLM response for code blocks and offer to execute them. Add a flag --execute that pipes the response through subprocess.run() after user confirmation. This is the gateway to "computer, run my test suite."

2. Conversation memory. Append each user/assistant exchange to the Ollama messages list. With a sliding window of the last 10 turns, you get context-aware follow-ups. Watch your token usage—Codestral's 32k context fills faster than you think.

3. Streaming TTS. Piper supports sentence-level streaming via its JSON output mode. Instead of waiting for the full LLM response, stream tokens from Ollama (stream=True), chunk on sentence boundaries, and pipe each chunk to Piper. The assistant starts speaking while the LLM is still generating—a massive perceived-latency win.

If you're building this as part of a portfolio to demonstrate forward-deployed engineering skills, the streaming TTS extension is particularly high-signal. It shows you understand buffer management, backpressure, and user-perceived performance. For more on what makes a portfolio project stand out, see our guide on The FDE Portfolio: What to Build to Get Hired and Stand Out from the Stack.

Common Pitfalls and Fixes

"Whisper transcribes gibberish." Your mic gain is too low or you're too far away. Check the RMS values in the callback—if they're consistently below 0.005, increase system mic gain. Also, ensure you're recording at 16kHz mono. Stereo or 44.1kHz will confuse Whisper's feature extraction.

"Ollama times out on first request." The model isn't loaded. Run ollama run codestral once in a terminal to preload it, or increase the timeout in the Python client. On slow disks, model loading can take 30+ seconds.

"Piper sounds robotic or choppy." You're likely feeding it text with special characters or markdown. The system prompt should forbid markdown. Also, Piper's --output-raw sample rate is 22050 Hz—if sd.play crackles, try sd.play(audio_np, samplerate=22050, blocksize=1024).

"The loop hangs after recording." Whisper is CPU-intensive and blocks the main thread. For a single-user tool this is fine, but if you need responsiveness during transcription, move transcribe_audio to a ThreadPoolExecutor. Don't prematurely optimize—profile first.

"My assistant responds to itself." If your speakers feed back into the mic, use headphones. Echo cancellation is a deep rabbit hole. For a terminal tool, push-to-talk with headphones solves it definitively.

This voice pipeline is the same pattern you'd use when embedding with a customer who needs an air-gapped assistant—no cloud dependencies, everything runs on-prem. If that deployment model sounds familiar, it's the core of How Palantir-Style FDEs Embed with Customers to Unlock Technical Value.

Frequently Asked Questions

Q: Can I use a different STT engine? Yes. Swap the transcribe_audio function for faster-whisper (CTranslate2 backend, lower latency) or Vosk (streaming, no file I/O). The interface is the same: bytes in, text out.

Q: How do I add a wake word like "Hey Computer"? Integrate openwakeword or Porcupine (free tier for personal use). Run the wake word detector on a continuous low-latency stream, and only invoke the full Whisper pipeline on detection. This increases CPU usage but removes the need to press Enter.

Q: Will this work on a Raspberry Pi? Whisper tiny runs on a Pi 4 with ~2x real-time factor (2 seconds to transcribe 1 second of audio). Piper runs comfortably. Ollama is the bottleneck—consider llama3.2:1b or phi3:mini for ARM. Expect 5-10 second total latency per query.

Q: Can I pipe the LLM output directly to a shell? Yes, but sanitize it. Never blind-execute LLM output. A safer pattern: have the LLM output structured JSON with a command field, validate it against a whitelist of allowed binaries, then subprocess.run with shell=False. This is production FDE thinking—automation without footguns.

Q: How does this compare to cloud APIs like ElevenLabs or OpenAI STT? Cloud APIs have lower latency and higher quality voices, but they cost money, require internet, and ship your audio off-machine. This build is free, private, and works on an airplane. The tradeoff is voice naturalness—Piper is good but not ElevenLabs-tier. For a deep dive on cost-performance tradeoffs in model selection, see DeepSeek V4 Flash 0731: Breaking Down the Latency, Throughput, and Cost Tradeoffs.

Q: I want to turn this into a deployable tool. What's next? Package it with pyinstaller for one-click distribution. Add a --daemon mode that sits in the system tray. Write integration tests that mock sounddevice and ollama. If you're thinking about how this fits into a broader engineering role where you ship prototypes directly with customers, our breakdown of What a Forward Deployed Engineer Actually Does in a Week: From Standup to Shipped Prototype maps directly to this kind of work.