Build Guides

Build a Competitor Monitoring Agent That Alerts on Site Changes Using Playwright

FDE Coach EditorialJuly 22, 202610 min read

What We're Building

A scheduled monitoring agent that scrapes competitor websites, detects content changes against a stored baseline, and pushes a crisp, human-readable summary into your Slack channel. No vendor lock-in, no monthly SaaS tax—just Playwright, a free LLM from Groq, Supabase for persistence, and the Slack API.

Feature list:

Headless browser scraping via Playwright (bypasses basic bot protection)
Baseline storage in Supabase (JSONB, free tier)
Semantic diff using Groq's Llama 3.3 70B (free tier, 30 requests/minute)
Slack Block Kit messages with change highlights
Cron-based scheduling (GitHub Actions free tier or local crontab)
Zero-cost operation under free-tier limits

This is the same pattern FDEs use when a customer asks, “Are our competitors shipping faster than us?” You wire up an agent, surface the signal, and skip the manual tab-refreshing.

Architecture Overview

Flow: Cron fires the script. Playwright fetches the target page, extracts structured text. The script pulls the last known baseline from Supabase. If no baseline exists, it stores one and exits. If a baseline exists, both old and new content hit Groq, which returns a concise diff summary. That summary lands in Slack via an incoming webhook. The new content becomes the next baseline.

Prerequisites (All Free-Tier)

Tool	Purpose	Free Tier Limit	Signup Link
Node.js 20+	Runtime	Unlimited	https://nodejs.org
Playwright	Browser automation	Unlimited	https://playwright.dev
Groq Cloud	LLM inference for diff summaries	30 req/min, ~14k req/day	https://console.groq.com
Supabase	Baseline storage (Postgres JSONB)	500 MB database, 2 projects	https://supabase.com
Slack Incoming Webhook	Alert delivery	Unlimited (within rate limits)	https://api.slack.com/messaging/webhooks
GitHub Actions (optional)	Free scheduling	2,000 min/month for private repos	https://github.com/features/actions

Grab API keys for Groq (GROQ_API_KEY) and Supabase (SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY). For Slack, create an app, enable incoming webhooks, and copy the webhook URL.

Step 1: Project Setup and Dependencies

mkdir competitor-monitor && cd competitor-monitor
npm init -y
npm install playwright @supabase/supabase-js groq-sdk dotenv
npx playwright install chromium

Create a .env file:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJ...
GROQ_API_KEY=gsk_...
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

Step 2: Baseline Capture Script

We'll build scraper.js—the core module that fetches a page and returns clean text. We target the <main> or <body> tag and strip scripts/styles.

// scraper.js
const { chromium } = require('playwright');

async function scrapePage(url) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  });
  const page = await context.newPage();
  
  try {
    await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });
    // Extract text from the main content area, fallback to body
    const content = await page.evaluate(() => {
      const main = document.querySelector('main') || document.body;
      // Remove script and style elements
      const clone = main.cloneNode(true);
      clone.querySelectorAll('script, style, noscript, nav, footer').forEach(el => el.remove());
      return clone.innerText.trim();
    });
    return content;
  } finally {
    await browser.close();
  }
}

module.exports = { scrapePage };

Now wire up Supabase. Run this SQL in the Supabase SQL Editor:

create table if not exists site_baselines (
  id serial primary key,
  url text not null unique,
  content text not null,
  captured_at timestamptz default now()
);

baseline.js handles storage and retrieval:

// baseline.js
const { createClient } = require('@supabase/supabase-js');
require('dotenv').config();

const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_SERVICE_ROLE_KEY
);

async function getBaseline(url) {
  const { data, error } = await supabase
    .from('site_baselines')
    .select('content')
    .eq('url', url)
    .single();
  
  if (error && error.code !== 'PGRST116') throw error;
  return data?.content || null;
}

async function setBaseline(url, content) {
  const { error } = await supabase
    .from('site_baselines')
    .upsert({ url, content, captured_at: new Date().toISOString() });
  
  if (error) throw error;
}

module.exports = { getBaseline, setBaseline };

Step 3: Diff and Summarize with Groq

This is where the free LLM shines. We send old and new content to Groq's Llama 3.3 70B and ask for a structured summary. The model is fast enough that the round-trip feels real-time.

// differ.js
const Groq = require('groq-sdk');
require('dotenv').config();

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });

async function summarizeDiff(oldContent, newContent, url) {
  // Truncate to avoid token limits (free tier context: 128k tokens, but keep it lean)
  const oldChunk = oldContent.slice(0, 15000);
  const newChunk = newContent.slice(0, 15000);

  const prompt = `You are a competitive intelligence analyst. Compare the OLD and NEW versions of a competitor's webpage.
Return a JSON object with:
- "has_changes": boolean
- "summary": a 2-3 sentence executive summary of what changed
- "key_changes": array of strings, each a bullet describing a specific change (max 5)
- "confidence": "high" | "medium" | "low"

URL: ${url}

---OLD CONTENT---
${oldChunk}
---END OLD---

---NEW CONTENT---
${newChunk}
---END NEW---

Return ONLY valid JSON, no markdown fences.`;

  const completion = await groq.chat.completions.create({
    messages: [{ role: 'user', content: prompt }],
    model: 'llama-3.3-70b-versatile',
    temperature: 0.1,
    response_format: { type: 'json_object' }
  });

  const result = JSON.parse(completion.choices[0].message.content);
  return result;
}

module.exports = { summarizeDiff };

Why response_format: { type: 'json_object' }? It forces the model to emit parseable JSON. Groq's Llama 3.3 supports this natively, and it eliminates regex-cleanup headaches.

Step 4: Slack Alerting

We format the Groq output into a Slack Block Kit message. This gives you rich formatting—bold headers, bulleted changes, and a direct link to the competitor page.

// alerter.js
async function sendSlackAlert(diffResult, url) {
  const webhookUrl = process.env.SLACK_WEBHOOK_URL;
  if (!diffResult.has_changes) {
    console.log(`No changes detected for ${url}, skipping alert.`);
    return;
  }

  const blocks = [
    {
      type: 'header',
      text: { type: 'plain_text', text: `🔍 Competitor Site Change: ${new URL(url).hostname}` }
    },
    {
      type: 'section',
      text: {
        type: 'mrkdwn',
        text: `*Summary:* ${diffResult.summary}\n\n*Confidence:* ${diffResult.confidence}`
      }
    },
    {
      type: 'section',
      text: {
        type: 'mrkdwn',
        text: '*Key Changes:*\n' + diffResult.key_changes.map(c => `• ${c}`).join('\n')
      }
    },
    {
      type: 'context',
      elements: [
        { type: 'mrkdwn', text: `<${url}|View live page> • Captured ${new Date().toISOString()}` }
      ]
    }
  ];

  await fetch(webhookUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ blocks })
  });
}

module.exports = { sendSlackAlert };

Step 5: Scheduling the Agent

Tie it all together in index.js:

// index.js
require('dotenv').config();
const { scrapePage } = require('./scraper');
const { getBaseline, setBaseline } = require('./baseline');
const { summarizeDiff } = require('./differ');
const { sendSlackAlert } = require('./alerter');

const TARGETS = [
  'https://competitor-one.com/pricing',
  'https://competitor-two.com/product'
];

async function monitor(url) {
  console.log(`Scraping ${url}...`);
  const newContent = await scrapePage(url);
  const oldContent = await getBaseline(url);

  if (!oldContent) {
    console.log(`No baseline for ${url}, storing initial snapshot.`);
    await setBaseline(url, newContent);
    return;
  }

  console.log(`Diffing ${url}...`);
  const diff = await summarizeDiff(oldContent, newContent, url);
  
  if (diff.has_changes) {
    console.log(`Changes detected for ${url}, sending alert.`);
    await sendSlackAlert(diff, url);
  } else {
    console.log(`No significant changes for ${url}.`);
  }

  // Update baseline regardless—we always want the latest state
  await setBaseline(url, newContent);
}

(async () => {
  for (const url of TARGETS) {
    await monitor(url);
  }
})();

Run it manually:

node index.js

For scheduling, drop a GitHub Actions workflow at .github/workflows/monitor.yml:

name: Competitor Monitor
on:
  schedule:
    - cron: '0 8,16 * * 1-5'  # 8 AM and 4 PM, weekdays
  workflow_dispatch:  # manual trigger

jobs:
  monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install chromium --with-deps
      - run: node index.js
        env:
          SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
          SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }}
          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Store the four secrets in your repo's Settings > Secrets and Variables > Actions. The workflow_dispatch trigger lets you kick off a manual run to test.

Extensions

Visual diff screenshots. Playwright can capture full-page screenshots. Store them in Supabase Storage (free 1 GB) and include the image URL in the Slack message. The free tier covers plenty of screenshots.

Multi-page crawling. Expand TARGETS with a sitemap parser. Use Playwright to extract all internal links from a competitor's nav, then diff every page. Watch your Groq rate limit—batch requests with a 2-second delay.

Pricing-specific monitoring. If you're tracking SaaS pricing pages, add a second Groq call that extracts structured pricing data (plan name, price, features) and stores it in a Supabase table. Then you can query pricing history over time.

Email digests. Swap Slack for a weekly email summary using Resend's free tier (100 emails/day). Aggregate all changes into one message and send it every Friday.

If you want to go deeper on agent patterns, the Build a GitHub Issue Triager That Auto-Labels and Routes to the Right Owner guide covers a similar orchestration loop with LLM-based routing. For turning raw content into structured assets, check out Build a YouTube-to-Blog Repurposing Agent Using Whisper and a Free LLM.

Common Pitfalls

Bot detection. Some sites serve different content to headless browsers. If you get empty or generic pages, rotate user agents, add --disable-blink-features=AutomationControlled to Chromium args, or use Playwright's stealth plugin (playwright-extra + puppeteer-extra-plugin-stealth).

Token limits. The summarizeDiff function truncates to 15k characters per chunk. For very long pages, consider splitting by <section> or sending multiple Groq calls and merging results. Monitor your Groq usage dashboard—the free tier is generous but not infinite.

Noise from dynamic content. Timestamps, cookie banners, and random IDs trigger false positives. Add a preprocessing step that strips ISO dates and UUIDs before diffing. A simple regex: content.replace(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/g, '[DATE]').

Supabase row limits. The free tier includes 500 MB. One baseline per URL is tiny, but if you store full-page screenshots as base64 in JSONB, you'll hit limits fast. Use Supabase Storage for binary assets.

FAQ

Q: Why Groq instead of OpenAI's free tier? A: Groq's free tier gives you Llama 3.3 70B at ~300 tokens/second—much faster than GPT-3.5 Turbo. The response_format: json_object support means no parsing hacks. Rate limits (30 req/min) are fine for scheduled monitoring.

Q: Can I monitor JavaScript-heavy SPAs? A: Yes. Playwright's waitUntil: 'networkidle' waits for XHR/fetch calls to settle. For React sites that hydrate slowly, add await page.waitForSelector('.main-content') with a specific CSS selector before extracting text.

Q: What if the competitor blocks my IP? A: GitHub Actions runners use shared IP ranges that most sites won't block. For aggressive protection, route through a free proxy (Webshare offers 10 free proxies) or use Playwright's proxy option in browser.newContext().

Q: How do I monitor paywalled or login-gated pages? A: Use Playwright to fill login forms before navigation. Store credentials in GitHub Secrets. Example: await page.fill('#email', process.env.COMPETITOR_EMAIL); await page.fill('#password', process.env.COMPETITOR_PASSWORD); await page.click('button[type="submit"]');. This is a gray area—review the site's ToS.

Q: Can I run this on my own server instead of GitHub Actions? A: Absolutely. Add a crontab entry: 0 8,16 * * 1-5 cd /path/to/project && node index.js >> /var/log/monitor.log 2>&1. No dependency on GitHub.

Building agents that turn raw internet data into actionable signals is core to the FDE skillset. This pattern—scrape, store, diff, alert—repeats across customer engagements. Master it once, reuse it everywhere.

#competitive-intel#web-scraping#monitoring#playwright

Want to build like a Forward Deployed Engineer?

FDE Coach is a cohort-based program in frontend, backend, AWS, and AI. Build real products and get referred to 200+ hiring partners.

Explore the program