Build a Personal Meeting Notetaker That Transcribes and Summarizes Calls

What We’re Building

A local CLI tool that sits on your machine, grabs system audio during a call, and spits out a clean Markdown summary. No cloud uploads, no per-minute pricing traps. The pipeline: capture loopback audio → transcribe with Whisper → feed the transcript to Groq’s fast inference endpoint (Mixtral or Llama 3) → get back a structured brief with decisions, action items, and owners.

Feature list:

• System audio capture (speaker + mic loopback) on macOS/Linux/Windows
• Near-real-time transcription via OpenAI Whisper (base or small model)
• Batch summarization with Groq’s free-tier LLM
• Markdown output with sections: Summary, Key Decisions, Action Items, Owners
• Zero-cost operation within free tiers

If you’ve ever left a call knowing something was decided but not who owns it, this tool fixes that. It’s also a clean demo of orchestrating local ML and a remote LLM API—a pattern that shows up constantly in Forward Deployed work. For more on shipping prototypes fast under constraints, see /blog/fde-customer-prototype-week-playbook.

Architecture Overview

Audio flows from the OS loopback device into a Python capture thread. That thread writes raw PCM to a queue. A Whisper worker pulls chunks, runs inference locally, and appends text to a buffer. When the call ends (or on demand), the accumulated transcript hits Groq’s chat completions endpoint with a summarization prompt. The response is rendered to a Markdown file. Everything except the Groq call runs locally.

Prerequisites (All Free Tier)

• Python 3.10+ — your machine likely has it; if not, brew install python or apt install python3
• pip for package management
• OpenAI Whisper — pip install openai-whisper (MIT license, runs on CPU or CUDA)
• Groq API key — sign up at console.groq.com (free tier gives generous requests/min for Mixtral-8x7b and Llama 3 8B/70B)
• sounddevice — pip install sounddevice for cross-platform audio capture
• numpy — pip install numpy (already pulled by Whisper)
• BlackHole (macOS) or VB-Cable (Windows) or PulseAudio loopback (Linux) for capturing system audio without physical loopback cables

Install system audio loopback:

• macOS: brew install blackhole-2ch
• Windows: download VB-Cable (free single-cable version)
• Linux: pactl load-module module-loopback latency_msec=1

Set your system’s output device to the loopback sink, and configure your meeting app (Zoom, Meet, Teams) to also output to that sink. Then your capture script reads from the loopback’s input side.

Step 1: Capturing System Audio with Python

We’ll use sounddevice because it abstracts away CoreAudio, WASAPI, and PulseAudio. The trick is finding the correct device index for the loopback input.

# audio_capture.py
import sounddevice as sd
import numpy as np
import queue
import threading

def list_devices():
    print(sd.query_devices())

def find_loopback_device():
    devices = sd.query_devices()
    for i, dev in enumerate(devices):
        if 'blackhole' in dev['name'].lower() or 'cable' in dev['name'].lower() or 'loopback' in dev['name'].lower():
            if dev['max_input_channels'] > 0:
                return i
    raise RuntimeError("No loopback device found. Install BlackHole/VB-Cable.")

def capture_audio(device_index, sample_rate=16000, block_duration=2.0, audio_queue=None):
    """Continuously capture audio blocks and put them in a queue."""
    def callback(indata, frames, time, status):
        if status:
            print(f"Audio status: {status}")
        audio_queue.put(indata.copy())

    with sd.InputStream(
        device=device_index,
        channels=1,
        samplerate=sample_rate,
        blocksize=int(sample_rate * block_duration),
        callback=callback
    ):
        print("Recording... Press Ctrl+C to stop.")
        try:
            while True:
                sd.sleep(100)
        except KeyboardInterrupt:
            print("Stopping capture.")

Key decisions: 16 kHz mono is plenty for Whisper and keeps CPU load low. A 2-second block size balances latency against transcription overhead. The queue decouples capture from processing—critical if Whisper inference takes longer than a block duration.

Step 2: Real-Time or Post-Call Transcription with Whisper

Whisper runs locally. The base model (142M params) is the sweet spot: fast enough for near-real-time on a modern laptop, accurate enough for meeting speech. If you have a GPU, small is even better.

# transcriber.py
import whisper
import numpy as np
import threading

def transcribe_worker(audio_queue, transcript_buffer, model_name="base"):
    """Pull audio blocks from queue, transcribe, append to buffer."""
    model = whisper.load_model(model_name)
    print(f"Whisper {model_name} model loaded.")

    while True:
        audio_block = audio_queue.get()
        if audio_block is None:  # sentinel to stop
            break

        # Whisper expects float32 in [-1, 1]
        audio_float = audio_block.flatten().astype(np.float32)
        # Normalize if needed (most loopback devices output float already)
        if np.max(np.abs(audio_float)) > 1.0:
            audio_float = audio_float / 32768.0

        result = model.transcribe(audio_float, fp16=False, language="en")
        text = result['text'].strip()
        if text:
            transcript_buffer.append(text)
            print(f"[Transcribed] {text[:80]}...")

For post-call (batch) mode, just accumulate all audio blocks into one big numpy array and run model.transcribe() once. Real-time mode is more impressive but batch mode gives slightly better accuracy because Whisper can use broader context.

Step 3: Structured Summarization via Groq LLM

Groq’s API is OpenAI-compatible, which means the openai Python library works directly. Groq’s free tier currently offers Mixtral-8x7b-32768 and Llama 3 8B/70B at high throughput. We’ll use Mixtral for its strong instruction-following.

# summarizer.py
from openai import OpenAI
import os

def summarize_transcript(transcript, api_key=None):
    if api_key is None:
        api_key = os.environ.get("GROQ_API_KEY")
    if not api_key:
        raise ValueError("Set GROQ_API_KEY environment variable or pass api_key.")

    client = OpenAI(
        base_url="https://api.groq.com/openai/v1",
        api_key=api_key
    )

    system_prompt = """You are a precise meeting summarizer. Given a raw transcript, produce a structured Markdown summary with these sections:

## Summary
(2-3 sentences capturing the meeting's purpose and outcome)

## Key Decisions
- Decision 1
- Decision 2

## Action Items
- [ ] Task description — **Owner: Name** (due: date if mentioned)

## Open Questions
- Question 1

Rules:
- Extract owner names from the transcript when stated (e.g., "Alice will handle X").
- If no owner is named, write **Owner: Unassigned**.
- Be concise. Do not hallucinate facts not in the transcript.
- If the transcript is fragmented or unclear, note that in the summary."""

    response = client.chat.completions.create(
        model="mixtral-8x7b-32768",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Meeting transcript:\n\n{transcript}"}
        ],
        temperature=0.2,
        max_tokens=1024
    )

    return response.choices[0].message.content

Why temperature=0.2? We want deterministic, factual extraction—not creative prose. This is a structured data extraction task masquerading as summarization. If you’re curious about tuning sampling parameters for different tasks, /blog/controlling-reasoning-effort-llm-inference goes deeper on the cost/accuracy tradeoff.

Step 4: Gluing It Together into a CLI Tool

Here’s the main script that wires everything together. It runs capture and transcription in separate threads, waits for Ctrl+C, then summarizes.

# meeting_notetaker.py
import argparse
import queue
import threading
import time
from audio_capture import find_loopback_device, capture_audio
from transcriber import transcribe_worker
from summarizer import summarize_transcript

def main():
    parser = argparse.ArgumentParser(description="Personal Meeting Notetaker")
    parser.add_argument("--output", "-o", default="meeting_summary.md", help="Output Markdown file")
    parser.add_argument("--model", default="base", choices=["tiny", "base", "small", "medium"], help="Whisper model size")
    parser.add_argument("--batch", action="store_true", help="Batch mode: transcribe after recording stops")
    args = parser.parse_args()

    device_idx = find_loopback_device()
    print(f"Using audio device {device_idx}")

    audio_queue = queue.Queue()
    transcript_buffer = []

    # Start capture thread
    capture_thread = threading.Thread(
        target=capture_audio,
        args=(device_idx, 16000, 2.0, audio_queue),
        daemon=True
    )
    capture_thread.start()

    if args.batch:
        # Batch mode: collect all audio, transcribe at end
        all_audio = []
        print("Recording (batch mode). Press Ctrl+C to stop and transcribe...")
        try:
            while True:
                block = audio_queue.get(timeout=0.5)
                all_audio.append(block.flatten())
        except KeyboardInterrupt:
            pass
        import numpy as np
        import whisper
        full_audio = np.concatenate(all_audio)
        model = whisper.load_model(args.model)
        result = model.transcribe(full_audio, fp16=False, language="en")
        transcript = result['text']
        print(f"\nTranscript ({len(transcript)} chars):\n{transcript[:200]}...")
    else:
        # Real-time mode
        transcribe_thread = threading.Thread(
            target=transcribe_worker,
            args=(audio_queue, transcript_buffer, args.model),
            daemon=True
        )
        transcribe_thread.start()
        print("Recording (real-time mode). Press Ctrl+C to stop...")
        try:
            while True:
                time.sleep(0.5)
        except KeyboardInterrupt:
            pass
        audio_queue.put(None)  # signal transcriber to stop
        transcribe_thread.join(timeout=5)
        transcript = " ".join(transcript_buffer)

    if not transcript.strip():
        print("No speech detected. Exiting.")
        return

    print("\nSending to Groq for summarization...")
    summary = summarize_transcript(transcript)

    with open(args.output, "w") as f:
        f.write(f"# Meeting Summary\n\nGenerated: {time.strftime('%Y-%m-%d %H:%M')}\n\n")
        f.write(summary)

    print(f"Summary saved to {args.output}")

if __name__ == "__main__":
    main()

Running the Notetaker

1. Set your system audio to route through the loopback device.
2. Set your meeting app’s speaker output to the same loopback device.
3. Export your Groq key: export GROQ_API_KEY="gsk_yourkey"
4. Run: python meeting_notetaker.py --output standup_notes.md
5. Join your call. Press Ctrl+C when it ends.
6. Open the Markdown file. You’ll have a clean summary with action items.

For real-time mode (default), you’ll see transcription snippets as they arrive. Batch mode (--batch) transcribes once at the end—more accurate but no live feedback.

Extensions Worth Building

• Speaker diarization: Use pyannote.audio (free for research) to label who said what. The summarizer prompt then gets speaker-labeled transcripts, making owner extraction far more accurate.
• Slack/Notion integration: After summary generation, POST the Markdown to a Slack webhook or append to a Notion page. A 20-line addition with requests.
• Hotword wake: Only transcribe when speech is detected (VAD with silero-vad). Saves CPU and avoids Whisper hallucinating on silence.
• Streaming to Groq: Instead of sending the full transcript, stream it chunk-by-chunk for a running summary that updates mid-call. Groq’s speed makes this viable.
• Local LLM fallback: If Groq is down, fall back to a local Ollama model. The pattern is identical—just swap the base URL. See /blog/codebase-qa-tool-with-ollama-and-llamaindex for a similar local-first architecture.

Common Pitfalls

1. Wrong audio device. Run python -c "import sounddevice; print(sounddevice.query_devices())" and confirm your loopback device shows max_input_channels > 0. If it’s 0, you’re looking at the output side.
2. Silent recordings. Your meeting app might not be routing to the loopback. On macOS, option-click the sound menu bar icon to verify. On Windows, check per-app output in Sound settings.
3. Whisper OOM on CPU. The medium and large models need significant RAM. Stick with base or small unless you have 16GB+ and patience.
4. Groq rate limits. Free tier has RPM limits. If you hit them, add a time.sleep(2) retry or cache transcripts locally and batch-summarize later.
5. Transcript too long for context window. Mixtral has a 32k context window—enough for ~2 hours of dense conversation. If you exceed it, chunk the transcript and summarize each chunk, then summarize the summaries.
6. Hallucinated action items. The LLM sometimes invents owners. The prompt’s “Do not hallucinate” instruction helps, but always review the output. For production pipelines, consider a verification step. The pattern is similar to debugging in constrained environments—/blog/fde-customer-zero-trust-debugging-playbook covers that mindset.

FAQ

Q: Does this work with Zoom/Google Meet/Microsoft Teams? A: Yes. Any app that lets you choose an audio output device works. Route that output to your loopback device, and the notetaker captures everything—both your voice (via mic loopback) and others’ voices (via speaker loopback).

Q: Is this really free? A: Whisper runs locally (free). Groq’s free tier gives enough requests per minute for dozens of meetings daily. The only cost is electricity. No API keys beyond Groq’s are needed.

Q: Can I use a different LLM? A: Absolutely. Swap the base_url and model in summarizer.py. Ollama, Anthropic, or OpenAI all work. Groq is chosen for speed and free tier generosity.

Q: How accurate is Whisper base model? A: On clear meeting audio, ~95% word accuracy. Accents and overlapping speech degrade it. The small model buys you another 2-3% accuracy at 2x the inference time.

Q: What about privacy? A: Audio never leaves your machine until the transcript hits Groq’s API. If that’s a concern, run a local LLM via Ollama for the summarization step too. The architecture supports it with a one-line base URL change.

Q: Can this run on a Raspberry Pi? A: Whisper tiny model runs on a Pi 4, but expect ~5-10x real-time factor. Not suitable for live calls, but fine for post-call batch processing of short meetings.

Q: How do I add speaker labels? A: Integrate pyannote.audio for diarization. It’s a separate model that assigns speaker IDs to audio segments. Then prepend [Speaker 1]: to each Whisper segment before building the transcript. The summarizer prompt already handles owner extraction from text patterns.

If you want to go deeper on shipping tools like this in a professional context—where the prototype becomes the product in a week—/blog/fde-customer-prototype-week-playbook walks through the full playbook.