Build an On-Call Incident Summarizer from Logs with Groq and Playwright

What We're Building

On-call engineers drown in alerts. You get paged, you scramble through Datadog, Grafana, or a raw log stream, and then you spend an hour writing a postmortem that nobody reads. We're going to automate the worst part: the first draft.

This bot scrapes your monitoring dashboards (even the ones behind a login) and raw log files, then feeds everything to Groq's free-tier Llama 3 70B to produce a clear, structured incident summary. You paste the logs, it gives you a draft postmortem.

Feature list:

• Headless browser scraping of any web-based dashboard (Grafana, Datadog, custom) using Playwright
• Local log file ingestion with smart sampling (you don't need 50k lines of INFO)
• Context-aware summarization via Groq's Llama 3 70B (free tier, 30 requests/min)
• Outputs a markdown postmortem with timeline, impact assessment, root cause hypothesis, and action items
• Optional: voice-to-text incident notes via Whisper (free, local)

Architecture Overview

This is a pipeline, not a service. A Python script pulls data from two sources, concatenates context, ships it to Groq, and writes a markdown file.

The scraper authenticates to your dashboard, grabs relevant panels or query results. The log parser filters for ERROR/WARN lines within a time window. The aggregator bundles this with a system prompt that forces the LLM to think like an SRE. Groq returns the draft. You review and ship.

Prerequisites

All free. No credit card required to start.

Grab your Groq API key from the console, export it:

export GROQ_API_KEY="gsk_your_key_here"

Step 1: Scraping Dashboards with Playwright

Most monitoring dashboards require authentication. Playwright handles this natively with persistent browser contexts. We'll log in once, save the state, and reuse it.

Create scraper.py:

import asyncio
from playwright.async_api import async_playwright
import json

AUTH_FILE = "auth_state.json"

async def login_and_save_state(url: str, username: str, password: str):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)  # Headful for initial login
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto(url)
        await page.fill('input[name="username"]', username)
        await page.fill('input[name="password"]', password)
        await page.click('button[type="submit"]')
        await page.wait_for_url("**/dashboards/**")  # Adjust to your post-login URL pattern
        await context.storage_state(path=AUTH_FILE)
        await browser.close()

async def scrape_dashboard(url: str, selector: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(storage_state=AUTH_FILE)
        page = await context.new_page()
        await page.goto(url, wait_until="networkidle")
        # Wait for the specific panel to render
        await page.wait_for_selector(selector, timeout=15000)
        content = await page.inner_text(selector)
        await browser.close()
        return content

Why this works: The storage_state file serializes cookies and local storage. On subsequent runs, you're already authenticated. For Grafana, your selector might be .panel-content. For Datadog, target the specific widget's container. If your dashboard is public or internal without auth, skip the login step entirely.

Pro tip: If the dashboard renders data via API calls, intercept the network responses instead of scraping DOM text. Playwright's page.route() lets you capture JSON payloads directly—cleaner data, less brittle selectors.

Step 2: Parsing Log Files

Dumping 50MB of logs into an LLM context wastes tokens and dilutes signal. We need aggressive sampling: errors, warnings, and lines immediately surrounding them.

Create log_parser.py:

import re
from datetime import datetime, timedelta
from typing import List, Tuple

def extract_incident_window(log_path: str, incident_time: str, window_minutes: int = 30) -> str:
    """
    Pulls ERROR/WARN lines within a time window around the incident.
    Assumes timestamps like: 2024-01-15 14:23:45,123
    """
    incident_dt = datetime.strptime(incident_time, "%Y-%m-%d %H:%M:%S")
    start_dt = incident_dt - timedelta(minutes=window_minutes)
    end_dt = incident_dt + timedelta(minutes=window_minutes)
    
    relevant_lines: List[str] = []
    timestamp_pattern = re.compile(r'^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}')
    
    with open(log_path, 'r') as f:
        lines = f.readlines()
    
    for i, line in enumerate(lines):
        match = timestamp_pattern.match(line)
        if not match:
            continue
        try:
            line_dt = datetime.strptime(match.group(), "%Y-%m-%d %H:%M:%S")
        except ValueError:
            continue
        
        if start_dt <= line_dt <= end_dt:
            if any(level in line for level in ['ERROR', 'WARN', 'FATAL', 'CRITICAL']):
                # Grab context: 3 lines before and after
                start_idx = max(0, i - 3)
                end_idx = min(len(lines), i + 4)
                relevant_lines.append(f"--- context around line {i} ---")
                relevant_lines.extend(lines[start_idx:end_idx])
    
    if not relevant_lines:
        return "No significant log events found in the incident window."
    
    # Truncate if too large (Llama 3 70B context is 8k tokens, but we share with dashboard data)
    max_chars = 6000
    result = "".join(relevant_lines)
    if len(result) > max_chars:
        result = result[:max_chars] + "\n... [truncated]"
    
    return result

This isn't a log aggregation tool—it's a triage scraper. If you're already shipping logs to Loki or Elasticsearch, query those APIs directly instead. The principle is the same: narrow the blast radius before the LLM sees it.

Step 3: Crafting the LLM Prompt for Groq

The difference between a garbage summary and a useful one is the system prompt. We need to force structure, insist on honesty about uncertainty, and prevent hallucination of details not present in the input.

Create prompt.py:

SYSTEM_PROMPT = """You are an SRE writing an internal incident postmortem draft. Your analysis must be evidence-based. Do not invent metrics, times, or root causes not present in the provided context.

Structure your response exactly as follows:

## Incident Summary
- **Start Time:** [extract or state "unknown"]
- **Detection Method:** [e.g., PagerDuty alert, manual report]
- **Duration:** [calculated or "unknown"]
- **Severity:** [SEV1/SEV2/SEV3 based on impact]

## Impact
- What user-facing symptoms were observed?
- What services were affected?

## Timeline (UTC)
| Time | Event |
|------|-------|
| ... | ... |

## Root Cause Hypothesis
- Leading hypothesis based on available evidence
- Confidence level (High/Medium/Low)
- Alternative hypotheses if evidence is ambiguous

## Action Items
- [ ] Immediate remediation (if not already done)
- [ ] Investigation follow-ups
- [ ] Prevention measures

## Open Questions
- What do we still not know?

If the provided context is insufficient, clearly state what additional data you would need."""

def build_messages(dashboard_data: str, log_data: str, incident_notes: str = "") -> list:
    user_prompt = f"""## Dashboard State at Time of Incident
{dashboard_data}

## Relevant Log Excerpts
{log_data}

## Engineer's Notes
{incident_notes if incident_notes else "No additional notes provided."}

Draft the postmortem."""
    
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_prompt}
    ]

The prompt forces a timeline table and explicit confidence levels. This is critical—LLMs love to sound certain. By demanding a confidence label, you get a more honest draft.

Step 4: Assembling the Incident Summarizer

Now the glue. Create summarizer.py:

import os
from groq import Groq
from scraper import scrape_dashboard, login_and_save_state
from log_parser import extract_incident_window
from prompt import build_messages
import argparse

def main():
    parser = argparse.ArgumentParser(description="On-Call Incident Summarizer")
    parser.add_argument("--dashboard-url", required=True, help="URL of the monitoring dashboard")
    parser.add_argument("--dashboard-selector", required=True, help="CSS selector for the dashboard panel")
    parser.add_argument("--log-path", required=True, help="Path to log file")
    parser.add_argument("--incident-time", required=True, help="Incident time as 'YYYY-MM-DD HH:MM:SS'")
    parser.add_argument("--notes", default="", help="Optional path to voice notes transcript")
    parser.add_argument("--output", default="postmortem_draft.md", help="Output markdown file")
    parser.add_argument("--login-url", default=None, help="Dashboard login URL (if auth required)")
    parser.add_argument("--username", default=None)
    parser.add_argument("--password", default=None)
    args = parser.parse_args()
    
    # Optional: login if first run
    if args.login_url and args.username and args.password:
        import asyncio
        asyncio.run(login_and_save_state(args.login_url, args.username, args.password))
        print("Auth state saved. Subsequent runs will reuse it.")
    
    # Scrape dashboard
    dashboard_data = asyncio.run(scrape_dashboard(args.dashboard_url, args.dashboard_selector))
    print(f"Scraped {len(dashboard_data)} chars from dashboard.")
    
    # Parse logs
    log_data = extract_incident_window(args.log_path, args.incident_time)
    print(f"Extracted log window: {len(log_data)} chars.")
    
    # Load optional voice notes
    incident_notes = ""
    if args.notes:
        with open(args.notes, 'r') as f:
            incident_notes = f.read()
    
    # Call Groq
    client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
    messages = build_messages(dashboard_data, log_data, incident_notes)
    
    print("Drafting postmortem with Llama 3 70B...")
    completion = client.chat.completions.create(
        model="llama3-70b-8192",
        messages=messages,
        temperature=0.3,  # Lower temperature for factual consistency
        max_tokens=4096,
    )
    
    draft = completion.choices[0].message.content
    
    with open(args.output, 'w') as f:
        f.write(draft)
    
    print(f"Postmortem draft written to {args.output}")

if __name__ == "__main__":
    main()

Running the Bot

First run (with auth):

python summarizer.py \
  --login-url "https://your-grafana-instance.com/login" \
  --username "oncall-bot@company.com" \
  --password "$DASHBOARD_PASSWORD" \
  --dashboard-url "https://your-grafana-instance.com/d/abc123/service-dashboard" \
  --dashboard-selector ".panel-container" \
  --log-path "/var/log/service.log" \
  --incident-time "2024-01-15 14:23:00" \
  --notes "incident_notes.txt"

Subsequent runs (reuses auth_state.json):

python summarizer.py \
  --dashboard-url "https://your-grafana-instance.com/d/abc123/service-dashboard" \
  --dashboard-selector ".panel-container" \
  --log-path "/var/log/service.log" \
  --incident-time "2024-01-15 14:23:00"

The output postmortem_draft.md lands in your working directory. Review it, fill in the blanks the LLM flagged, and ship it to your incident management tool.

Sensible Extensions

1. Voice-to-text incident notes with Whisper. While you're fighting the fire, record a voice memo. Whisper (free, local) transcribes it:

pip install openai-whisper
whisper incident_notes.mp3 --model base --output_format txt

Feed that transcript via --notes. The LLM will incorporate your real-time observations.

2. PagerDuty webhook trigger. Instead of running manually, wire this to a PagerDuty incident trigger. When an incident is acknowledged, a webhook fires the script with the incident time and service name. Now you have a draft waiting before you even open your laptop.

3. Multi-dashboard correlation. Scrape multiple dashboards (app metrics, DB metrics, CDN) and pass all of them into the context. The LLM will cross-reference and produce a richer analysis.

4. Historical pattern matching. Store past postmortems in a vector DB. When a new incident occurs, retrieve the top 3 similar past incidents and include them in the prompt as reference. This turns the bot from a summarizer into an institutional memory.

If you're thinking about building more automation into your on-call workflow, the Build a Gmail AI Triage Agent That Drafts Replies Using Gemini and Groq pattern is directly applicable—replace email triage with alert triage and the architecture is nearly identical.

Common Pitfalls

Selector brittleness. Dashboard UIs change. Your .panel-content selector works today, breaks tomorrow. Mitigation: use data attributes if your dashboard tool supports them ([data-testid="cpu-panel"]). Better yet, intercept the underlying API calls instead of scraping DOM.

Auth state expiration. The auth_state.json contains session cookies that expire. If your scraper suddenly returns login pages, delete the file and re-authenticate. For production, use API keys or service accounts instead of user/password login.

Context window overflow. Llama 3 70B has an 8k context window. If you dump raw dashboard HTML plus 10k lines of logs, you'll get truncated output or an API error. The log parser's hard truncation at 6000 chars plus dashboard data keeps us safe, but monitor your total prompt size.

Groq rate limits. Free tier is 30 requests per minute. During a major incident, you might run the script multiple times as new data arrives. Add a simple retry with backoff:

import time
try:
    completion = client.chat.completions.create(...)
except groq.RateLimitError:
    time.sleep(60)
    completion = client.chat.completions.create(...)

LLM overconfidence. Even with our prompt demanding confidence labels, Llama 3 will sometimes state hypotheses as facts. Always treat the output as a draft, not a finished document. The engineer in the loop is the most critical component.

FAQ

Q: Can I use a different LLM provider? A: Yes. Swap the Groq client for OpenAI, Anthropic, or a local Ollama instance. The prompt structure is provider-agnostic. Groq is chosen here because it's fast and has a generous free tier—critical when you need a draft in under 30 seconds during an incident.

Q: What if my logs don't have timestamps? A: The log parser relies on timestamps for windowing. If your logs use a different format, modify the regex in extract_incident_window. If they have no timestamps at all, you'll need to pass the entire file (with aggressive truncation) or switch to a log aggregation system.

Q: How do I handle dashboards that load data dynamically via JavaScript? A: Playwright's wait_for_selector with networkidle handles most SPA dashboards. If data loads incrementally, add explicit page.wait_for_timeout(2000) after navigation. For extreme cases, use page.evaluate() to poll for specific DOM elements.

Q: Is this production-ready? A: As a local CLI tool for on-call engineers, yes. As an automated service, it needs error handling, secret management (don't hardcode passwords), and monitoring. But the core pipeline—scrape, parse, summarize—is the same pattern used in production incident management at companies running LLMs in their on-call workflows.

---

Building this kind of automation is exactly the skillset that gets noticed in forward-deployed engineering roles. If you're looking to sharpen your ability to ship pragmatic AI tooling under real constraints, check out The FDE Portfolio: What to Build to Demonstrate Deployment Velocity and Get Hired for more project patterns that demonstrate the right instincts.