Build a Free Competitor Monitoring Agent with Playwright and Groq

What We're Building

A fully autonomous agent that monitors competitor websites for changes, diffs the content, and delivers concise AI-powered summaries straight to your terminal. No paid APIs, no credit card required.

Feature list:

• Headless browser scraping with Playwright (handles JavaScript-heavy sites)
• Content extraction and normalization (strips noise like ads and nav)
• Semantic diffing that ignores meaningless changes (timestamps, rotating testimonials)
• AI summarization via Groq's free Llama 3.1 tier (8,000 requests/day)
• Persistent storage of snapshots in Supabase's free tier
• Configurable monitoring targets via a simple JSON file

Architecture Overview

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  targets.json│────▶│ Orchestrator │────▶│  Playwright  │
│  (config)    │     │   (main.js)  │     │  Scraper     │
└──────────────┘     └──────┬───────┘     └──────┬───────┘
                            │                    │
                            ▼                    ▼
                     ┌──────────────┐     ┌──────────────┐
                     │   Supabase   │◀────│    Diff      │
                     │  (snapshots) │     │   Engine     │
                     └──────────────┘     └──────┬───────┘
                                                 │
                                                 ▼
                                          ┌──────────────┐
                                          │ Groq Llama   │
                                          │ Summarizer   │
                                          └──────────────┘

The orchestrator reads targets, fetches the previous snapshot from Supabase, scrapes the current page, computes a normalized diff, sends it to Groq for summarization, stores the new snapshot, and prints the alert.

Prerequisites

Everything here is free-tier. No credit card.

You'll need API keys for Supabase (project URL + anon key) and Groq (API key from console). Store them in a .env file—never hardcode.

Step 1: Project Setup and Dependencies

mkdir competitor-watchdog && cd competitor-watchdog
npm init -y
npm install playwright @supabase/supabase-js groq-sdk diff dotenv
npx playwright install chromium

Create your .env:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=eyJhbGciOi...
GROQ_API_KEY=gsk_your_key_here

And a targets.json:

[
  {
    "id": "acme-pricing",
    "url": "https://acme-corp.com/pricing",
    "selector": "main",
    "cron": "0 */6 * * *"
  },
  {
    "id": "rival-blog",
    "url": "https://competitor.io/blog",
    "selector": ".post-content",
    "cron": "0 0 * * *"
  }
]

The selector field tells Playwright which DOM element to extract. Use main for the primary content area, or a specific class/ID. The cron field is optional—we'll handle scheduling separately, but it's there for documentation.

Step 2: Database Schema with Supabase

Head to your Supabase SQL Editor and run:

CREATE TABLE snapshots (
  id BIGSERIAL PRIMARY KEY,
  target_id TEXT NOT NULL,
  captured_at TIMESTAMPTZ DEFAULT NOW(),
  content TEXT NOT NULL,
  content_hash TEXT NOT NULL
);

CREATE INDEX idx_snapshots_target_id_captured_at
  ON snapshots(target_id, captured_at DESC);

This schema stores the full scraped content and a SHA-256 hash for quick change detection. The index ensures fast lookups of the most recent snapshot per target.

Create a Supabase client module at lib/supabase.js:

import { createClient } from '@supabase/supabase-js';
import dotenv from 'dotenv';
dotenv.config();

export const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_ANON_KEY
);

Step 3: The Scraping Engine with Playwright

Create lib/scraper.js:

import { chromium } from 'playwright';

export async function scrapePage(url, selector = 'main') {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
  });
  const page = await context.newPage();

  try {
    // Block unnecessary resources for speed
    await page.route('**/*', (route) => {
      const type = route.request().resourceType();
      if (['image', 'stylesheet', 'font', 'media'].includes(type)) {
        route.abort();
      } else {
        route.continue();
      }
    });

    await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });

    // Wait for the selector to be present
    await page.waitForSelector(selector, { timeout: 10000 });

    const content = await page.$eval(selector, (el) => el.innerText);
    return normalizeText(content);
  } finally {
    await browser.close();
  }
}

function normalizeText(text) {
  return text
    .replace(/\s+/g, ' ')       // collapse whitespace
    .replace(/[\u200B-\u200D]/g, '') // zero-width chars
    .trim();
}

Key decisions: We block images and stylesheets to speed up scraping—we only care about text content. The networkidle wait ensures JavaScript-rendered content loads. The normalizer strips invisible characters that cause false-positive diffs.

Step 4: Diffing and Change Detection

Create lib/differ.js:

import { diffWords } from 'diff';
import crypto from 'crypto';

export function hashContent(content) {
  return crypto.createHash('sha256').update(content).digest('hex');
}

export function computeDiff(oldContent, newContent) {
  if (!oldContent) {
    return { hasChanged: true, summary: 'Initial snapshot. No previous data.' };
  }

  const changes = diffWords(oldContent, newContent);
  const added = changes.filter(c => c.added).map(c => c.value).join('');
  const removed = changes.filter(c => c.removed).map(c => c.value).join('');

  if (!added && !removed) {
    return { hasChanged: false, summary: 'No changes detected.' };
  }

  // Build a compact diff for the LLM
  let diffText = 'CHANGES DETECTED:\n';
  if (removed) diffText += `--- REMOVED: ${removed.slice(0, 500)}\n`;
  if (added) diffText += `+++ ADDED: ${added.slice(0, 500)}\n`;

  return { hasChanged: true, summary: diffText };
}

We use the diff library for word-level diffs—character-level is too noisy, line-level misses inline changes. We cap the diff at 500 characters per side to stay within Groq's context window and keep summarization fast.

Step 5: Summarization with Groq's Free Tier

Create lib/summarizer.js:

import Groq from 'groq-sdk';

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });

export async function summarizeChanges(diffText, targetId) {
  if (diffText === 'No changes detected.') return diffText;

  const prompt = `You are a competitive intelligence analyst. A website change was detected on "${targetId}".
Here is the raw diff (ADDED and REMOVED text):

${diffText}

Summarize what changed in 2-3 bullet points. Focus on:
- Pricing changes
- New features or product announcements
- Messaging or positioning shifts
- Team or leadership changes

Be concise. If the change is trivial (e.g., a typo fix), say so.`;

  try {
    const completion = await groq.chat.completions.create({
      messages: [{ role: 'user', content: prompt }],
      model: 'llama-3.1-8b-instant',
      temperature: 0.2,
      max_tokens: 200,
    });

    return completion.choices[0]?.message?.content || 'Summary unavailable.';
  } catch (err) {
    if (err.status === 429) {
      return 'Rate limited by Groq. Try again in a minute.';
    }
    throw err;
  }
}

We use llama-3.1-8b-instant—it's fast, free, and more than capable for summarization. Temperature is set low (0.2) for factual consistency. The 429 handler prevents crashes during rate limits.

Step 6: The Orchestrator Agent

Create main.js:

import { readFileSync } from 'fs';
import { scrapePage } from './lib/scraper.js';
import { supabase } from './lib/supabase.js';
import { hashContent, computeDiff } from './lib/differ.js';
import { summarizeChanges } from './lib/summarizer.js';

async function getPreviousSnapshot(targetId) {
  const { data } = await supabase
    .from('snapshots')
    .select('content')
    .eq('target_id', targetId)
    .order('captured_at', { ascending: false })
    .limit(1)
    .single();
  return data?.content || null;
}

async function storeSnapshot(targetId, content) {
  await supabase.from('snapshots').insert({
    target_id: targetId,
    content,
    content_hash: hashContent(content),
  });
}

async function monitorTarget(target) {
  console.log(`[${target.id}] Scraping ${target.url}...`);
  const newContent = await scrapePage(target.url, target.selector);

  console.log(`[${target.id}] Fetching previous snapshot...`);
  const oldContent = await getPreviousSnapshot(target.id);

  console.log(`[${target.id}] Computing diff...`);
  const diff = computeDiff(oldContent, newContent);

  if (diff.hasChanged) {
    console.log(`[${target.id}] Summarizing with Groq...`);
    const summary = await summarizeChanges(diff.summary, target.id);

    console.log(`\n📢 ALERT: ${target.id}\n${summary}\n`);
  } else {
    console.log(`[${target.id}] No changes.\n`);
  }

  await storeSnapshot(target.id, newContent);
}

async function main() {
  const targets = JSON.parse(readFileSync('./targets.json', 'utf-8'));

  for (const target of targets) {
    try {
      await monitorTarget(target);
    } catch (err) {
      console.error(`[${target.id}] Failed:`, err.message);
    }
  }
}

main().catch(console.error);

How to Run It

node main.js

Expected output:

[acme-pricing] Scraping https://acme-corp.com/pricing...
[acme-pricing] Fetching previous snapshot...
[acme-pricing] Computing diff...
[acme-pricing] Summarizing with Groq...

📢 ALERT: acme-pricing
- Enterprise tier price increased from $499/mo to $599/mo
- New "Startup" plan added at $79/mo with 5 seats
- Removed mention of 24/7 phone support from all tiers

For recurring runs, add a cron job:

# Run every 6 hours
0 */6 * * * cd /path/to/competitor-watchdog && node main.js >> logs.txt 2>&1

Or use a simple setInterval wrapper for a long-running process:

import { setInterval } from 'timers';
setInterval(main, 6 * 60 * 60 * 1000); // every 6 hours

Sensible Extensions

Slack/Email alerts: Pipe the summary to a webhook. Supabase has a free Edge Functions integration for this.

Multi-page funnels: Extend targets.json to accept an array of URLs per target, simulating a user journey. This catches changes in onboarding flows or checkout pages.

Screenshot diffs: Playwright can capture full-page screenshots. Store them in Supabase Storage and use pixel-diff libraries like pixelmatch for visual regression.

Scheduled Supabase Edge Function: Move the entire orchestrator to a Supabase Edge Function triggered by pg_cron. This eliminates the need for a server. See our Edge Function cron guide.

Competitor pricing table parser: Add a structured extraction step that parses pricing tables into JSON before diffing. This enables programmatic price tracking and trend analysis.

Common Pitfalls

Selector fragility: Sites change their DOM. If your selector breaks, the scraper throws. Use broad selectors like main or article and fall back to body if the primary selector fails.

Groq rate limits: The free tier is generous but finite. Implement exponential backoff if you're monitoring dozens of sites. A 30-second delay between targets usually suffices.

Memory leaks in long-running processes: Playwright launches a new browser per scrape. That's intentional—it prevents memory buildup. Don't reuse browser instances across hours-long intervals.

Content hash collisions: SHA-256 is safe, but if you're paranoid, store both the hash and a content length check. We already store full content, so you can always do a deep comparison.

IP blocking: Some sites block datacenter IPs. If you hit this, consider routing Playwright through a free proxy or using residential proxy services (though those break the "free" constraint).

FAQ

Q: How is this different from visualping or Distill.io? A: Those are SaaS products with monthly fees and limited free tiers. This is fully self-hosted, infinitely customizable, and genuinely free. You own the data and the pipeline.

Q: Can I monitor password-protected pages? A: Yes. Add authentication steps to the Playwright script—fill login forms, store cookies, or set headers. The scraper is just Playwright; you can script any browser interaction.

Q: What if the site is a single-page app (SPA)? A: That's why we use Playwright instead of fetch or cheerio. The networkidle wait handles SPAs. If you need to wait for specific API responses, use page.waitForResponse().

Q: How much does Supabase free tier actually hold? A: 500 MB of database storage. Text-only snapshots are tiny—you'll fit years of monitoring before hitting the limit. If you add screenshot storage, you'll fill it faster; use their Storage bucket with a retention policy.

Q: Can I deploy this on a Raspberry Pi? A: Absolutely. Playwright's Chromium runs on ARM. This entire setup sips resources—perfect for a Pi 4 sitting in a closet.

Q: What if Groq changes their free tier? A: The summarizer module is swappable. You can drop in Ollama for fully local summarization, or use another free-tier API like Together AI.