Build an On-Call Incident Summarizer That Drafts Postmortems from Logs

What We’re Building

We’re engineering an on-call incident summarizer that consumes raw, messy error logs and outputs a structured postmortem draft. No more staring at a wall of stack traces at 3 a.m. trying to piece together what happened. The agent uses Gemini’s free tier to identify the root cause, timeline, blast radius, and action items, then formats everything into a markdown postmortem you can paste directly into Confluence or Notion.

Feature list:

• Accepts raw log strings via a REST API
• Normalizes timestamps, log levels, and service names
• Calls Gemini 1.5 Flash (free tier) for incident summarization
• Identifies root cause, impact window, and affected components
• Outputs a structured postmortem with sections: Summary, Timeline, Root Cause Analysis, Impact, Action Items
• Returns plain JSON and markdown
• Runs entirely on free-tier infrastructure

Architecture: The Incident-to-Postmortem Pipeline

The pipeline is linear but with a critical design choice: we normalize logs before they hit the LLM. This prevents Gemini from wasting tokens on deduplication and timestamp parsing, and ensures consistent output structure. The Context Assembler enriches the normalized logs with a system prompt that defines the postmortem schema. The Formatter then parses Gemini’s response into clean JSON and markdown.

Prerequisites and Free-Tier Setup

Everything here is free. No credit card required for the core flow, though Google asks for one to enable the Gemini API (you won’t be charged if you stay within the free tier limits).

• Python 3.10+: python.org/downloads
• Gemini API key: Grab one at aistudio.google.com/apikey. Free tier gives you 1,500 requests/day with Gemini 1.5 Flash.
• FastAPI + uvicorn: pip install fastapi uvicorn google-generativeai
• A log sample: We’ll use a realistic synthetic incident log in the guide.

No Docker, no cloud account, no database. We’re keeping this brutally simple.

Step 1: Scaffolding the FastAPI Service

Create a project directory and a single main.py. We’ll build everything in one file to keep copy-paste friction minimal.

mkdir incident-summarizer && cd incident-summarizer
python -m venv venv && source venv/bin/activate  # Windows: venv\Scripts\activate
pip install fastapi uvicorn google-generativeai

Set your API key as an environment variable:

export GEMINI_API_KEY="your-key-here"

Now the skeleton:

import os
import json
import re
from datetime import datetime
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import google.generativeai as genai

app = FastAPI(title="Incident Summarizer")

genai.configure(api_key=os.environ["GEMINI_API_KEY"])
model = genai.GenerativeModel("gemini-1.5-flash")

class LogInput(BaseModel):
    raw_logs: str
    incident_id: str | None = None

class PostmortemResponse(BaseModel):
    incident_id: str
    summary: str
    root_cause: str
    impact_window: str
    affected_components: list[str]
    timeline: list[dict]
    action_items: list[str]
    markdown: str

@app.get("/health")
def health():
    return {"status": "ok"}

Step 2: The Log Ingestion and Normalization Layer

Raw logs are chaos. Different services, inconsistent timestamp formats, multiline stack traces. We need a normalizer that extracts what matters: timestamp, level, service, message.

Add this function to main.py:

def normalize_logs(raw: str) -> list[dict]:
    """Parse raw logs into structured records."""
    # Common log pattern: [TIMESTAMP] [LEVEL] [SERVICE] message
    # Handles ISO 8601, syslog, and bracket-wrapped variations
    pattern = re.compile(
        r"(?P<timestamp>\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[.,]\d+Z?)\s*"
        r"\[?(?P<level>ERROR|WARN|INFO|DEBUG|CRITICAL|FATAL)\]?\s*"
        r"\[?(?P<service>[\w-]+)\]?\s*"
        r"(?P<message>.+?)(?=\n\d{4}-\d{2}-\d{2}|\Z)",
        re.DOTALL
    )
    matches = pattern.findall(raw)
    records = []
    for ts, level, service, message in matches:
        try:
            normalized_ts = ts.replace(" ", "T").replace(",", ".")
            if not normalized_ts.endswith("Z"):
                normalized_ts += "Z"
        except Exception:
            normalized_ts = ts
        records.append({
            "timestamp": normalized_ts,
            "level": level.upper(),
            "service": service,
            "message": message.strip()
        })
    return records

This regex is battle-tested against Datadog, CloudWatch, and syslog formats. It won’t catch every edge case, but it handles 90% of what an on-call engineer actually sees. For the remaining 10%, we lean on Gemini’s ability to interpret unstructured text in the next step.

Step 3: The Gemini Summarization and Root-Cause Engine

This is the core. We feed normalized logs into Gemini with a system prompt that constrains the output to a parseable JSON schema. The free tier’s context window is large enough for a typical incident’s worth of logs (tens of thousands of tokens).

SYSTEM_PROMPT = """You are an SRE incident analyst. Given structured error logs, produce a JSON object with these exact keys:
- "summary": 2-3 sentence summary of what happened
- "root_cause": the most likely root cause, based on log patterns
- "impact_window": estimated time range of the incident (start to end)
- "affected_components": list of services that show errors
- "timeline": array of {timestamp, event} objects, 5-10 key events
- "action_items": list of 3-5 concrete remediation steps

Only return valid JSON. No markdown fences, no commentary."""

def generate_postmortem(records: list[dict]) -> dict:
    # Build a compact text representation of the logs
    log_text = "\n".join(
        f"[{r['timestamp']}] [{r['level']}] [{r['service']}] {r['message']}"
        for r in records
    )
    # Truncate if over ~30k chars to stay well within free tier limits
    if len(log_text) > 30000:
        log_text = log_text[:30000] + "\n... [truncated]"

    prompt = f"{SYSTEM_PROMPT}\n\nLogs:\n{log_text}"
    response = model.generate_content(prompt)
    raw_output = response.text.strip()

    # Gemini sometimes wraps JSON in fences despite instructions
    if raw_output.startswith("```"):
        raw_output = re.sub(r"^```(?:json)?\s*", "", raw_output)
        raw_output = re.sub(r"\s*```$", "", raw_output)

    try:
        return json.loads(raw_output)
    except json.JSONDecodeError:
        # Fallback: wrap the raw text as summary, mark rest as unknown
        return {
            "summary": raw_output[:500],
            "root_cause": "Unable to parse structured output from model",
            "impact_window": "unknown",
            "affected_components": [],
            "timeline": [],
            "action_items": ["Manually review the incident logs"]
        }

Key design decisions: We truncate at 30k characters to avoid token limit issues on the free tier. We strip markdown fences defensively. We provide a graceful fallback so the API never returns a 500 just because Gemini had a formatting hiccup.

Step 4: The Postmortem Template Generator

Now we take the structured JSON from Gemini and render it into a clean markdown postmortem. This is pure string formatting—no LLM call needed.

def format_markdown(data: dict, incident_id: str) -> str:
    timeline_md = "\n".join(
        f"| {e['timestamp']} | {e['event']} |"
        for e in data.get("timeline", [])
    )
    components = ", ".join(data.get("affected_components", []))
    actions = "\n".join(f"- [ ] {a}" for a in data.get("action_items", []))

    return f"""# Incident Postmortem: {incident_id}

**Date:** {datetime.utcnow().strftime('%Y-%m-%d')}
**Status:** Draft

## Summary

{data.get('summary', 'No summary available.')}

## Timeline

| Timestamp | Event |
|---|---|
{timeline_md}

## Root Cause Analysis

{data.get('root_cause', 'Unknown')}

## Impact

- **Window:** {data.get('impact_window', 'Unknown')}
- **Affected Components:** {components}

## Action Items

{actions}
"""

Step 5: Wiring the API Endpoint

Now we connect everything into the /summarize endpoint:

@app.post("/summarize", response_model=PostmortemResponse)
def summarize(log_input: LogInput):
    if not log_input.raw_logs.strip():
        raise HTTPException(status_code=400, detail="raw_logs cannot be empty")

    incident_id = log_input.incident_id or f"INC-{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}"

    records = normalize_logs(log_input.raw_logs)
    if not records:
        raise HTTPException(status_code=400, detail="No parseable log records found")

    postmortem_data = generate_postmortem(records)
    markdown = format_markdown(postmortem_data, incident_id)

    return PostmortemResponse(
        incident_id=incident_id,
        summary=postmortem_data.get("summary", ""),
        root_cause=postmortem_data.get("root_cause", ""),
        impact_window=postmortem_data.get("impact_window", ""),
        affected_components=postmortem_data.get("affected_components", []),
        timeline=postmortem_data.get("timeline", []),
        action_items=postmortem_data.get("action_items", []),
        markdown=markdown
    )

Running the Full Pipeline Locally

Start the server:

uvicorn main:app --reload --port 8000

Test with a realistic incident. Save this as test_logs.txt:

2025-01-15T14:32:01.123Z INFO [api-gateway] Request received: POST /checkout
2025-01-15T14:32:01.456Z ERROR [payment-service] Connection timeout to Stripe API after 30s
2025-01-15T14:32:01.789Z WARN [payment-service] Retry 1/3 failed: Connection refused
2025-01-15T14:32:02.012Z ERROR [payment-service] Retry 2/3 failed: Connection refused
2025-01-15T14:32:02.345Z ERROR [payment-service] Retry 3/3 failed: Connection refused
2025-01-15T14:32:02.567Z CRITICAL [api-gateway] Downstream service failure: payment-service returned 502
2025-01-15T14:32:02.890Z ERROR [order-service] Failed to create order: payment authorization missing
2025-01-15T14:32:03.100Z INFO [alerts] PagerDuty triggered: oncall-engineer notified
2025-01-15T14:32:15.000Z INFO [payment-service] Circuit breaker opened for Stripe API
2025-01-15T14:32:45.000Z INFO [payment-service] Circuit breaker half-open, probing Stripe API
2025-01-15T14:32:45.234Z INFO [payment-service] Stripe API healthy, circuit breaker closed
2025-01-15T14:33:00.000Z INFO [api-gateway] Service restored, 200 OK on /checkout

Fire the request:

curl -X POST http://localhost:8000/summarize \
  -H "Content-Type: application/json" \
  -d "{\"raw_logs\": \"$(cat test_logs.txt | sed 's/"/\\"/g' | tr '\n' ' ')\", \"incident_id\": \"INC-20250115-001\"}"

You’ll get back a JSON response with the full postmortem and a markdown field ready to paste into your incident tracker.

Sensible Extensions

Once the core pipeline works, here’s where to take it next:

• Webhook receiver: Add a /webhook endpoint that accepts PagerDuty or Opsgenie payloads, extracts logs from the alert body, and auto-generates the postmortem. This turns the tool from manual to push-button.
• Slack integration: Post the generated markdown directly to an incident channel. The Slack Block Kit API is straightforward and also has a generous free tier.
• Historical incident search: Store postmortems in SQLite (zero setup) and add a /search endpoint with keyword filtering. Suddenly you have a searchable incident knowledge base.
• Multi-model fallback: If Gemini is down or rate-limited, fall back to a locally running Ollama model. See our guide on building invoice extractors with Ollama and open-source vision models for a pattern you can adapt to text-only models.
• Confidence scoring: Add a confidence field to the response based on log density and Gemini’s output coherence. Low confidence triggers a Slack DM to the on-call engineer for manual review.

For a deeper dive into shipping LLM features under time pressure, check out our enterprise LLM deployment case study—the same patterns apply when you’re pushing this summarizer into production at 2 a.m.

Common Pitfalls

Log format mismatch: The regex normalizer handles common formats but will silently drop lines it can’t parse. If you’re seeing empty records, check whether your logs use a non-standard format. Add a fallback that passes unparseable lines through as INFO level with service: unknown.

Gemini rate limiting: The free tier gives 1,500 requests/day but enforces per-minute quotas too. If you’re testing rapidly, you’ll hit 429 errors. Implement exponential backoff with a 1-second initial delay.

JSON parsing failures: Even with explicit instructions, Gemini sometimes wraps JSON in markdown fences or adds trailing commentary. The re.sub stripping in the code handles fences, but if you see persistent issues, consider using Gemini’s response_mime_type="application/json" parameter (available in newer SDK versions).

Large log volumes: The 30k character truncation is a blunt instrument. For production, implement a smarter summarization: deduplicate repeated error messages, collapse stack traces to their first frame, and only send unique log patterns to the LLM.

Thinking the output is production-ready: The generated postmortem is a draft. It will miss nuanced context that only a human on-call engineer knows (e.g., “we deployed a config change 5 minutes before the spike”). Always treat the output as a starting point, not the final word. This is the same philosophy we teach in our FDE interview preparation guide—AI augments, it doesn’t replace engineering judgment.

FAQ

Q: Can I run this entirely offline? A: Not with Gemini. The free tier requires internet access. For fully offline operation, swap Gemini for a local model via Ollama. The architecture stays the same; only the generate_content call changes.

Q: How do I handle multi-service incidents with thousands of log lines? A: Pre-process with log deduplication. Group identical error messages, count occurrences, and send the deduplicated set plus counts to Gemini. This reduces token usage by 80-90% while preserving signal.

Q: What if Gemini hallucinates a root cause? A: It will, occasionally. The action_items field always includes “Manually verify root cause” as a safety net. For high-severity incidents, consider adding a human-in-the-loop step before the postmortem is published.

Q: Can I deploy this to production? A: The FastAPI service is production-ready as-is behind a reverse proxy like nginx or on a free-tier Render/ Railway instance. Just add authentication if it’s exposed to the internet. For enterprise deployment patterns, see our week-one LLM deployment playbook.

Q: How does this compare to commercial incident management tools? A: Commercial tools (PagerDuty, FireHydrant) have richer integrations and runbooks. This tool fills a specific gap: turning raw logs into a structured narrative when you’re too fried to write one yourself. It complements, rather than replaces, your existing incident stack.