Deploy a Free Competitor Site Monitor That Alerts on Meaningful Changes Only

What We're Building

We're deploying a scheduled competitor monitor that doesn't spam you with every pixel change. It scrapes a target webpage, computes a clean text diff against the last known state, and uses a free LLM (Groq) to classify whether the changes are business-relevant—a pricing tweak, a new feature launch, a leadership change—or just noise like timestamps and rotating testimonials. Only meaningful changes land in your inbox via Resend.

Feature list:

• Scheduled scraping with Playwright (headless Chromium)
• Text extraction that strips HTML, scripts, and style cruft
• Diff computation against a stored baseline
• Free LLM summarization that filters out cosmetic changes
• Email digest via Resend’s free tier (100 emails/day)
• Entirely serverless on Cloudflare Workers free tier

This is the kind of tool a Forward Deployed Engineer builds in an afternoon to give a sales or product team an unfair information advantage. If you want to see how FDEs turn a messy customer problem into a shipped prototype in a week, check out this breakdown.

Architecture: How the Pieces Fit

The flow is linear but stateful. Cloudflare Workers KV acts as our cheap, persistent store for the previous snapshot of each monitored page. On each run, we scrape fresh content, diff it against KV, and pass the diff to Groq. Groq’s free tier gives us enough tokens to run this daily on several pages. If Groq returns a meaningful summary, we fire an email through Resend.

Prerequisites (All Free Tier)

• Cloudflare account – Workers free tier includes 100k requests/day and KV storage. Sign up.
• Groq API key – Free tier gives generous rate limits on fast inference. Get yours at console.groq.com.
• Resend API key – Free tier: 100 emails/day. resend.com/signup.
• Node.js 18+ and Wrangler CLI: npm install -g wrangler

We’ll use Playwright’s official Cloudflare Workers binding (@cloudflare/playwright), which runs a managed browser. No need to provision your own infra.

Step 1: Scaffold a Cloudflare Worker

mkdir competitor-monitor && cd competitor-monitor
npx wrangler init

Choose “Hello World” Worker, TypeScript. This generates wrangler.toml and src/index.ts.

Edit wrangler.toml to bind KV and set the compatibility date:

name = "competitor-monitor"
main = "src/index.ts"
compatibility_date = "2024-12-01"

[[kv_namespaces]]
binding = "SNAPSHOTS"
id = "your-kv-namespace-id"

Create the KV namespace:

npx wrangler kv:namespace create "SNAPSHOTS"

Paste the returned id into wrangler.toml.

Add secrets for the API keys (never hardcode):

npx wrangler secret put GROQ_API_KEY
npx wrangler secret put RESEND_API_KEY

Step 2: Write the Playwright Scraper

We’ll use Cloudflare’s Playwright binding. Install the package:

npm install @cloudflare/playwright

In src/index.ts, start with the browser launch and page fetch:

import { Browser } from '@cloudflare/playwright';

export interface Env {
  SNAPSHOTS: KVNamespace;
  GROQ_API_KEY: string;
  RESEND_API_KEY: string;
  BROWSER: Browser;
}

interface MonitorConfig {
  url: string;
  name: string;
  selector?: string; // optional CSS selector to scope scraping
}

const TARGETS: MonitorConfig[] = [
  { name: 'competitor-home', url: 'https://competitor.com', selector: 'main' },
  // add more entries here
];

Now the scraping function:

async function scrapePage(
  browser: Browser,
  config: MonitorConfig
): Promise<string> {
  const context = await browser.newContext();
  const page = await context.newPage();
  
  try {
    await page.goto(config.url, { waitUntil: 'networkidle', timeout: 30000 });
    
    let text: string;
    if (config.selector) {
      text = await page.locator(config.selector).innerText({ timeout: 10000 });
    } else {
      text = await page.locator('body').innerText({ timeout: 10000 });
    }
    
    // Normalize whitespace
    return text.replace(/\s+/g, ' ').trim();
  } finally {
    await context.close();
  }
}

The selector parameter is clutch. Scraping the entire body often pulls in nav footers and cookie banners that change constantly. Scoping to a main or article element gives a cleaner signal.

Step 3: Compute and Store a Text Diff

We need a lightweight diff. No heavy libraries—just a line-level comparison.

function computeDiff(oldText: string, newText: string): string {
  if (!oldText) return `[NEW PAGE]\n${newText.slice(0, 2000)}`;
  
  const oldLines = oldText.split('\n');
  const newLines = newText.split('\n');
  const added: string[] = [];
  const removed: string[] = [];
  
  const oldSet = new Set(oldLines);
  const newSet = new Set(newLines);
  
  for (const line of newLines) {
    if (!oldSet.has(line)) added.push(line);
  }
  for (const line of oldLines) {
    if (!newSet.has(line)) removed.push(line);
  }
  
  let diff = '';
  if (removed.length) diff += `--- REMOVED ---\n${removed.join('\n').slice(0, 1000)}\n\n`;
  if (added.length) diff += `+++ ADDED +++\n${added.join('\n').slice(0, 2000)}`;
  
  return diff || 'No significant text changes detected.';
}

This set-based approach is O(n) and works well for HTML text extraction where line ordering can shift. It’s not a proper Myers diff, but it catches additions and removals reliably.

Store and retrieve snapshots from KV:

async function getStoredSnapshot(env: Env, key: string): Promise<string | null> {
  return env.SNAPSHOTS.get(key);
}

async function storeSnapshot(env: Env, key: string, text: string): Promise<void> {
  await env.SNAPSHOTS.put(key, text);
}

Step 4: Summarize Meaningful Changes with Groq

The magic step. We send the diff to Groq’s free tier with a prompt that forces it to ignore cosmetic noise. We’ll use llama-3.1-8b-instant for speed and zero cost.

interface GroqResponse {
  choices: { message: { content: string } }[];
}

async function summarizeChanges(
  diff: string,
  pageName: string,
  apiKey: string
): Promise<string | null> {
  if (diff === 'No significant text changes detected.') return null;
  
  const prompt = `You are an analyst monitoring competitor websites for business-relevant changes.

Below is a text diff from scraping "${pageName}".

Rules:
- Ignore changes in timestamps, dates, cookie consent text, and rotating testimonials.
- Only report changes that indicate: pricing updates, new product features, leadership changes, partnership announcements, funding news, or strategic messaging shifts.
- If nothing is business-relevant, respond with exactly "NO_MEANINGFUL_CHANGE".
- If meaningful, write a 2-3 sentence summary of what changed and why it matters.

DIFF:
${diff.slice(0, 4000)}`;

  const response = await fetch('https://api.groq.com/openai/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'llama-3.1-8b-instant',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 300,
      temperature: 0.1,
    }),
  });

  const data = (await response.json()) as GroqResponse;
  const content = data.choices[0]?.message?.content?.trim() || '';
  
  if (content === 'NO_MEANINGFUL_CHANGE') return null;
  return content;
}

We cap the diff at 4000 characters to stay well within Groq’s free context window. The low temperature (0.1) keeps the output deterministic—we want classification, not creativity.

Step 5: Send the Email Digest with Resend

Resend’s API is dead simple. We’ll send a plain-text email with the summary.

async function sendEmail(
  summary: string,
  pageName: string,
  url: string,
  apiKey: string,
  recipient: string
): Promise<void> {
  await fetch('https://api.resend.com/emails', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      from: 'Competitor Monitor <monitor@yourdomain.com>',
      to: [recipient],
      subject: `[Change Detected] ${pageName}`,
      text: `A meaningful change was detected on ${pageName} (${url}):\n\n${summary}\n\n---\nThis is an automated alert from your competitor monitor.`,
    }),
  });
}

You’ll need to verify a sending domain in Resend’s dashboard. For testing, use the sandbox domain they provide.

Step 6: Wire Up the Cron Trigger

The Worker’s fetch handler becomes our cron entrypoint. Add a scheduled handler in wrangler.toml:

[triggers]
crons = ["0 8 * * *"]

This runs daily at 8 AM UTC. Now the main orchestrator:

export default {
  async scheduled(
    _event: ScheduledEvent,
    env: Env,
    _ctx: ExecutionContext
  ): Promise<void> {
    const browser = env.BROWSER;
    const recipient = 'you@yourdomain.com'; // hardcode or use env var
    
    for (const target of TARGETS) {
      try {
        console.log(`Scraping ${target.name}...`);
        const freshText = await scrapePage(browser, target);
        const oldText = (await getStoredSnapshot(env, target.name)) || '';
        
        const diff = computeDiff(oldText, freshText);
        console.log(`Diff length: ${diff.length} chars`);
        
        const summary = await summarizeChanges(diff, target.name, env.GROQ_API_KEY);
        
        if (summary) {
          console.log(`Meaningful change detected: ${summary.slice(0, 100)}...`);
          await sendEmail(summary, target.name, target.url, env.RESEND_API_KEY, recipient);
        } else {
          console.log('No meaningful change.');
        }
        
        // Always store the fresh snapshot for next comparison
        await storeSnapshot(env, target.name, freshText);
      } catch (err) {
        console.error(`Failed processing ${target.name}:`, err);
      }
    }
  },
};

We store the new snapshot after sending the email, so the alert references the correct baseline. Errors on one target don’t block the rest.

How to Run It

1. Test locally (Playwright won’t work fully in local dev; we test the logic):

   npx wrangler dev --test-scheduled
   

   Then hit http://localhost:8787/__scheduled to trigger.

2. Deploy:

   npx wrangler deploy
   

3. Verify the cron: Check the Cloudflare Dashboard → Workers → your worker → Triggers. You’ll see the cron schedule listed.

4. First run behavior: The initial scrape has no baseline, so computeDiff will return a [NEW PAGE] prefix. The LLM will see this and likely flag it. That’s fine—it seeds your first snapshot. Subsequent runs will diff normally.

Sensible Extensions

This is a solid v1. Here’s where you take it next:

• Multi-page monitoring: Add more entries to the TARGETS array. Each gets its own KV key automatically.
• Slack/Discord webhook: Swap Resend for a webhook if your team lives in chat. The pattern is identical.
• Screenshot diffing: Capture a full-page screenshot with page.screenshot({ fullPage: true }), store as base64 in KV, and compare structural changes visually. Groq’s vision models (like LLaVA) can analyze screenshot diffs if you upgrade later.
• Change history: Store the last N snapshots in KV with timestamped keys (snapshot:competitor-home:2024-12-01) for a rudimentary audit trail.
• Confidence scoring: Have Groq return a JSON object with meaningful: boolean and confidence: number. Only email above a threshold.

For a deeper look at how FDEs think about metrics like time-to-value and adoption velocity when shipping tools like this, see this breakdown of FDE metrics.

Common Pitfalls

FAQ

Q: Why not use a visual diffing tool like Percy? A: Those tools flag every CSS tweak. We want business intelligence, not QA alerts. The LLM filter is the entire point.

Q: Can I monitor paywalled or login-gated pages? A: Yes. Add a page.fill() step before goto to log in, and store credentials as Cloudflare secrets. Be mindful of the target’s terms of service.

Q: What if Groq hallucinates a summary? A: At temperature: 0.1, hallucination is minimal. The prompt constrains output tightly. If you’re paranoid, add a second LLM call for verification, but that burns free tier tokens.

Q: How many pages can I monitor on the free tier? A: Cloudflare Workers gives 100k requests/day. A daily cron on 10 pages uses 10 requests. Groq’s free tier handles ~30 requests/minute. You’re comfortably within limits for dozens of pages.

Q: Where can I learn more about building scraper-based automation? A: Check out how to auto-rewrite your resume using Playwright and free LLMs—the browser automation patterns are directly transferable.

Q: What’s the FDE angle here? A: This is classic FDE work: stitch together free APIs into a tool that gives a business team an edge, ship it in an afternoon, and iterate based on their feedback. If you’re curious what that week looks like end-to-end, here’s a time-study breakdown of an FDE’s week.