Build a Terminal Voice Assistant with Open-Source Whisper, Piper TTS, and Groq

What We're Building

A terminal-native voice assistant that runs entirely on your machine—no cloud STT/TTS costs, no proprietary lock-in. You speak into your mic, the tool transcribes your words with OpenAI’s open-source Whisper model, ships the text to Groq’s free-tier Llama 3 for reasoning, and pipes the LLM’s response through Piper TTS to read the answer aloud. Think of it as a local, privacy-respecting, hackable Siri for your command line.

Feature list:

• Push-to-talk voice recording with minimal latency
• Local, offline-capable transcription via Whisper (base model fits on a potato)
• Free LLM inference via Groq’s LPU cloud—blazing fast token generation
• Natural-sounding speech synthesis with Piper, no GPU required
• Single Python script orchestrator with clean error handling
• Fully free-tier: no API keys that expire into a billing surprise

Architecture: The Voice Pipeline

Before we write a line of code, let’s map the data flow. This is a classic linear pipeline with four stages, each replaceable if you want to swap models later.

The beauty of this design is its modularity. Each block is a standalone function with a clear input/output contract. When Piper releases a better voice, you swap one function. When you want to experiment with a different LLM on Groq, you change the model string. This is the kind of extensible software design that keeps side projects from collapsing under their own weight after the second feature request.

Prerequisites (All Free-Tier)

You need four things installed. Everything is free and open-source.

Groq free tier limits: 30 requests per minute, 14,400 per day on Llama 3 8B. Plenty for personal use. No credit card required at signup.

Piper voice model: Download a .onnx + .json pair from the Piper voice releases. I recommend en_US-lessac-medium for a natural American English voice. Put both files in a piper_models/ directory.

Step 1: Scaffolding the Project

Create a directory and a virtual environment. We’re keeping dependencies minimal.

mkdir terminal-voice-assistant
cd terminal-voice-assistant
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

pip install openai-whisper pyaudio groq numpy

Create the project structure:

terminal-voice-assistant/
├── main.py              # Orchestrator
├── recorder.py          # Audio capture
├── transcriber.py       # Whisper wrapper
├── llm.py               # Groq client
├── speaker.py           # Piper TTS wrapper
├── piper_models/        # Your downloaded .onnx + .json
│   ├── en_US-lessac-medium.onnx
│   └── en_US-lessac-medium.onnx.json
└── .env                 # GROQ_API_KEY=your_key_here

Load environment variables. Install python-dotenv or just export the key:

pip install python-dotenv

Step 2: Recording Voice Input

We’ll use PyAudio to capture audio until the user releases a key. Push-to-talk avoids transcribing background noise.

Create recorder.py:

import pyaudio
import wave
import numpy as np

CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000  # Whisper expects 16kHz
SILENCE_THRESHOLD = 500  # Adjust based on your mic
SILENCE_DURATION = 1.5   # Seconds of silence to auto-stop

def record_audio(filename="input.wav", max_duration=30):
    """Record audio until silence or max duration. Returns file path."""
    p = pyaudio.PyAudio()
    stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE,
                    input=True, frames_per_buffer=CHUNK)

    print("🎤 Recording... (speak now, pause 1.5s to stop)")
    frames = []
    silent_chunks = 0
    max_chunks = int(RATE / CHUNK * max_duration)

    for i in range(max_chunks):
        data = stream.read(CHUNK, exception_on_overflow=False)
        frames.append(data)

        # Simple silence detection on raw amplitude
        audio_chunk = np.frombuffer(data, dtype=np.int16)
        if np.abs(audio_chunk).mean() < SILENCE_THRESHOLD:
            silent_chunks += 1
        else:
            silent_chunks = 0

        if silent_chunks > int(RATE / CHUNK * SILENCE_DURATION):
            print("Silence detected, stopping.")
            break

    stream.stop_stream()
    stream.close()
    p.terminate()

    # Save to WAV
    wf = wave.open(filename, 'wb')
    wf.setnchannels(CHANNELS)
    wf.setsampwidth(p.get_sample_size(FORMAT))
    wf.setframerate(RATE)
    wf.writeframes(b''.join(frames))
    wf.close()
    return filename

Why 16kHz mono? Whisper resamples anyway, but feeding it native 16kHz avoids an extra conversion step and reduces file size. The silence detection is crude but effective—you can replace it with WebRTC VAD later if you need precision.

Step 3: Transcribing with Open-Source Whisper

Whisper runs locally. The base model is ~142MB and runs on CPU in under a second for short utterances. If you have a GPU, swap to small or medium.

Create transcriber.py:

import whisper

# Load once at module level—this is slow, do it at startup
_model = None

def get_model(model_name="base"):
    global _model
    if _model is None:
        print(f"Loading Whisper model '{model_name}'...")
        _model = whisper.load_model(model_name)
    return _model

def transcribe(audio_path, model_name="base"):
    """Transcribe audio file. Returns text string."""
    model = get_model(model_name)
    result = model.transcribe(audio_path, fp16=False)
    text = result["text"].strip()
    print(f"📝 Transcription: {text}")
    return text

Key detail: fp16=False keeps inference on CPU. If you have a CUDA-capable GPU, Whisper will use it automatically, but the base model is so small that CPU is fine for interactive use.

Step 4: Processing with Groq (Llama 3)

Groq provides a free, OpenAI-compatible API endpoint. We’ll send the transcription as a user message with a system prompt that keeps responses concise—nobody wants TTS reading a novel.

Create llm.py:

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

SYSTEM_PROMPT = """You are a helpful, concise terminal assistant. Keep responses under 3 sentences unless the user asks for detail. Be direct. No pleasantries, no markdown."""

def process_command(user_text, model="llama3-8b-8192"):
    """Send user text to Groq, return LLM response."""
    chat_completion = client.chat.completions.create(
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_text}
        ],
        model=model,
        temperature=0.7,
        max_tokens=150,  # Keep it tight for voice
    )
    response = chat_completion.choices[0].message.content.strip()
    print(f"🤖 Response: {response}")
    return response

Why max_tokens=150? Piper TTS reads at ~150 words per minute. A 150-token response is about 20-30 seconds of speech—long enough to be useful, short enough to not feel like a lecture. Adjust based on your patience.

Step 5: Speaking the Response with Piper TTS

Piper is a fast neural TTS engine that runs locally. We call it as a subprocess because the Python bindings are still maturing; the CLI is rock-solid.

Create speaker.py:

import subprocess
import os

PIPER_BINARY = "piper"  # Assumes piper is on PATH; otherwise use absolute path
MODEL_PATH = os.path.join("piper_models", "en_US-lessac-medium.onnx")
OUTPUT_FILE = "output.wav"

def speak(text):
    """Synthesize text to speech and play it."""
    if not text:
        return

    # Piper reads from stdin, writes WAV to stdout
    cmd = [PIPER_BINARY, "--model", MODEL_PATH, "--output_file", OUTPUT_FILE]
    proc = subprocess.run(cmd, input=text, text=True, capture_output=True)

    if proc.returncode != 0:
        print(f"Piper error: {proc.stderr}")
        return

    # Play the audio (platform-specific)
    if os.name == "posix":
        subprocess.run(["aplay", OUTPUT_FILE])  # Linux
    else:
        subprocess.run(["afplay", OUTPUT_FILE])  # macOS

Cross-platform playback: The snippet above covers macOS (afplay) and Linux (aplay). On Windows, replace with powershell -c (New-Object Media.SoundPlayer 'output.wav').PlaySync(). If you want a cleaner cross-platform solution, pip install playsound and call playsound.playsound(OUTPUT_FILE)—it’s a 3-line change.

Step 6: The Main Orchestration Loop

Now we wire everything together. A simple loop: record → transcribe → process → speak → repeat until the user says “exit” or “quit.”

Create main.py:

import os
import time
from dotenv import load_dotenv
from recorder import record_audio
from transcriber import transcribe
from llm import process_command
from speaker import speak

load_dotenv()

EXIT_PHRASES = {"exit", "quit", "goodbye", "stop"}

def main():
    print("🚀 Terminal Voice Assistant ready.")
    print("   Say 'exit' to quit.\n")

    while True:
        try:
            # 1. Record
            audio_path = record_audio()

            # 2. Transcribe
            user_text = transcribe(audio_path)
            if not user_text:
                print("Nothing heard, listening again...")
                continue

            # 3. Check for exit
            if user_text.lower() in EXIT_PHRASES:
                speak("Goodbye!")
                print("👋 Exiting.")
                break

            # 4. Process with LLM
            response = process_command(user_text)

            # 5. Speak response
            speak(response)

            # Small pause before next recording
            time.sleep(0.5)

        except KeyboardInterrupt:
            print("\n👋 Interrupted. Exiting.")
            break
        except Exception as e:
            print(f"❌ Error: {e}")
            speak("Sorry, I hit an error. Try again.")

if __name__ == "__main__":
    main()

How to Run It

1. Ensure Piper is on your PATH or update PIPER_BINARY in speaker.py.
2. Place your downloaded Piper voice model in piper_models/.
3. Export your Groq key or create a .env file:

   echo 'GROQ_API_KEY=gsk_your_key_here' > .env
   

4. Run the assistant:

   python main.py
   

5. Speak after the “Recording…” prompt. Pause 1.5 seconds to auto-stop, or press Ctrl+C to interrupt.

First run will download the Whisper base model (~142MB). Subsequent runs load it from cache.

Sensible Extensions

Once the basic loop works, you’ll itch to improve it. Here’s where to invest your next 90 minutes:

• Streaming TTS: Piper can stream audio. Pipe its stdout directly to a player instead of writing to disk. This cuts perceived latency by 40-60% because playback starts before synthesis finishes.
• Wake word detection: Integrate Porcupine (free for personal use) or OpenWakeWord so you can say “Hey Terminal” instead of push-to-talk. This transforms the tool from a utility into an ambient assistant.
• Tool use: Add a function-calling layer. If the user says “What’s the weather in Berlin?”, Groq returns a structured tool call, your Python fetches from wttr.in, and the result feeds back into the LLM for a spoken summary. This is the exact pattern forward deployed engineers use when embedding LLMs into customer workflows.
• Conversation memory: Append the last N exchanges to the Groq messages array. Now you have context-aware follow-ups like “What about tomorrow?” without re-explaining the topic.
• Model hot-swap: Add a --model CLI flag to switch between llama3-8b-8192, mixtral-8x7b-32768, or gemma-7b-it on Groq’s free tier. Different models have different personalities—find yours.

Common Pitfalls and Fixes

“Whisper loads every time I run.” The module-level cache in transcriber.py fixes this, but only if you call get_model() once at startup. If you’re spawning subprocesses, the model reloads. Keep everything in one process.

“Piper sounds robotic.” You probably downloaded the low quality voice. Grab a medium variant—the difference is night and day, and the ONNX file is still under 50MB.

“Groq returns 429 Too Many Requests.” You’re hitting the free-tier rate limit (30 RPM). Add a time.sleep(2) between requests or batch your prompts. The limit resets every minute.

“Silence detection cuts me off mid-sentence.” Your mic gain is too low, or you pause naturally. Increase SILENCE_DURATION to 2.0 seconds or lower SILENCE_THRESHOLD. Better yet, replace the amplitude-based VAD with WebRTC VAD (pip install webrtcvad)—it’s a 20-line swap and dramatically more robust.

“PyAudio installation fails on macOS M1/M2.” This is a known PortAudio headache. The reliable fix:

brew install portaudio
pip install --global-option='build_ext' --global-option='-I/opt/homebrew/include' --global-option='-L/opt/homebrew/lib' pyaudio

FAQ

Q: Can this run fully offline? A: The STT (Whisper) and TTS (Piper) stages are completely local. Only the LLM inference requires internet to reach Groq’s API. If you want 100% offline, swap Groq for a local model via Ollama (llama3.2:3b runs on a laptop), but expect slower token generation.

Q: What’s the end-to-end latency? A: On a modern laptop with the base Whisper model: ~0.8s transcription + ~0.3s Groq inference + ~0.5s Piper synthesis = roughly 1.6 seconds from end of speech to start of response audio. Streaming TTS can shave another 0.3s.

Q: Why not use the Groq Whisper endpoint instead of local Whisper? A: Groq does offer hosted Whisper, and it’s fast. But local Whisper costs zero API calls, works offline, and keeps your audio on your machine. For a privacy-first tool, local STT is the right call. You’re already sending text to Groq—no need to send raw audio too.

Q: How does this relate to what an FDE actually builds? A: This pipeline—sensor input → model inference → action output—is the same pattern used in enterprise deployments where FDEs wire up legacy systems to LLMs. The tools differ, but the architecture is identical. Building voice assistants teaches you latency budgeting, model selection, and pipeline error handling that translate directly to customer-facing systems.

Q: Can I use this in a production app? A: The free Groq tier is for development and personal use. For production, you’d need a paid plan. But the Whisper + Piper local pipeline is production-ready today—both are used in Home Assistant, Rhasspy, and other open-source voice platforms.

Q: Does this work on a Raspberry Pi? A: Yes, with the tiny Whisper model and Piper’s low quality voice. Expect 3-5 second latency. The Pi 5 handles it comfortably; a Pi 4 will struggle with base Whisper but manages tiny. This is a great weekend project that turns a Pi into a dedicated voice assistant appliance.