Build a Free Lead Enrichment Agent from Domains (OpenRouter + Playwright)

What We're Building

A CLI agent that takes a CSV of domain names, scrapes each company's website with Playwright, extracts meaningful text, then pipes it to OpenRouter's free models for summarization and ICP (Ideal Customer Profile) scoring. Results land in a Supabase table you can query, export, or hook into your CRM.

Feature list:

• Ingest a CSV of domains (one per line or column)
• Headful or headless scraping with Playwright (handles JS-heavy sites)
• Text extraction and cleaning (boilerplate removal)
• AI summarization via OpenRouter's free models (e.g., Mistral 7B, Gemma)
• ICP scoring with structured JSON output (firmographics, signals, fit score)
• Upsert results into Supabase (free tier, 500 MB database)
• Resumable: skip already-processed domains

This isn't a toy—by the end you'll have a production-adjacent pipeline that costs $0 to run.

Architecture Overview

• Playwright handles navigation, waits for content, and extracts inner text. We use the playwright npm package (free, open-source).
• OpenRouter provides a unified API to free models. We'll use mistralai/mistral-7b-instruct:free or google/gemma-7b-it:free. Rate limits apply, but fine for batch processing.
• Supabase stores results. We'll create a leads table with columns for domain, summary, ICP score, and raw JSON. Free tier includes 500 MB and 50,000 rows—plenty for thousands of leads.

See how this compares to scraping with Firecrawl if you want a managed alternative.

Prerequisites (All Free Tier)

1. Node.js 18+ – Download
2. Playwright – npm init -y && npm i playwright
3. OpenRouter API key – Sign up, create a key, free credits on signup. Free models have rate limits (~20 req/min).
4. Supabase account – Free tier, create a project, note your project URL and anon key.
5. CSV of domains – a file domains.csv with a header domain (e.g., stripe.com, vercel.com).

If you prefer Python, check Python scraping with Playwright for a similar setup.

Step 1: Project Setup and Environment

Create a new directory and install dependencies:

mkdir lead-enrichment-agent
cd lead-enrichment-agent
npm init -y
npm install playwright @supabase/supabase-js openai csv-parse dotenv

Notice we install openai – OpenRouter is OpenAI-compatible, so we use the official client pointed at https://openrouter.ai/api/v1.

Create a .env file:

OPENROUTER_API_KEY=sk-or-v1-...
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=eyJhbGciOi...

Create index.js—this will be our main pipeline. Also create a lib/ folder for scraper, enricher, and db modules.

Step 2: Scraping with Playwright

File: lib/scraper.js

const { chromium } = require('playwright');

async function scrapeDomain(domain) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
  });
  const page = await context.newPage();

  try {
    const url = domain.startsWith('http') ? domain : `https://${domain}`;
    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 15000 });
    // Wait for body text to appear
    await page.waitForSelector('body', { timeout: 10000 });
    
    // Extract all visible text
    const text = await page.evaluate(() => document.body.innerText);
    // Basic cleaning: collapse whitespace, take first 3000 chars
    const cleaned = text.replace(/\s+/g, ' ').trim().slice(0, 3000);
    return { domain, text: cleaned, url };
  } catch (err) {
    console.error(`Failed to scrape ${domain}:`, err.message);
    return { domain, text: '', url: `https://${domain}`, error: err.message };
  } finally {
    await browser.close();
  }
}

module.exports = { scrapeDomain };

Key decisions:

• domcontentloaded is faster than networkidle; most content is present.
• 3000-character limit keeps prompt size small for free models.
• Browser closes after each domain to avoid memory leaks.

For more robust scraping (proxies, retries), see Playwright best practices.

Step 3: Enrichment with OpenRouter Free Models

File: lib/enricher.js

const OpenAI = require('openai');

const openai = new OpenAI({
  baseURL: 'https://openrouter.ai/api/v1',
  apiKey: process.env.OPENROUTER_API_KEY,
  defaultHeaders: {
    'HTTP-Referer': 'http://localhost:3000', // Required by OpenRouter
    'X-Title': 'LeadEnrichmentAgent'
  }
});

async function enrich(domain, text) {
  if (!text || text.length < 50) {
    return { summary: 'Insufficient content', icp: {} };
  }

  const prompt = `Analyze this company website text and return a JSON object with:
- "summary": a 2-3 sentence description of what the company does
- "icp": an object with:
    - "industry": string
    - "size": "small" | "medium" | "large"
    - "b2b": boolean
    - "likely_decision_maker": string (job title)
    - "fit_score": number 1-10 (how likely they need B2B SaaS tools)

Text: ${text.substring(0, 2500)}

Return ONLY valid JSON, no markdown fences.`;

  try {
    const completion = await openai.chat.completions.create({
      model: 'mistralai/mistral-7b-instruct:free',
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.1,
      max_tokens: 400
    });

    const raw = completion.choices[0].message.content;
    // Strip possible markdown fences
    const jsonStr = raw.replace(/```json/g, '').replace(/```/g, '').trim();
    return JSON.parse(jsonStr);
  } catch (err) {
    console.error(`Enrichment failed for ${domain}:`, err.message);
    return { summary: 'Enrichment error', icp: {} };
  }
}

module.exports = { enrich };

Model choice: mistralai/mistral-7b-instruct:free is reliable and fast. If rate-limited, swap to google/gemma-7b-it:free. Both are free on OpenRouter.

We enforce JSON output via prompt engineering. For production, you'd use function calling or structured outputs—but free models often lack that. See structured output with LLMs for advanced patterns.

Step 4: Storing Results in Supabase

First, create the leads table in your Supabase SQL editor:

CREATE TABLE leads (
  id SERIAL PRIMARY KEY,
  domain TEXT UNIQUE NOT NULL,
  summary TEXT,
  icp JSONB,
  raw_text TEXT,
  url TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

File: lib/db.js

const { createClient } = require('@supabase/supabase-js');

const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_ANON_KEY
);

async function upsertLead({ domain, summary, icp, text, url }) {
  const { data, error } = await supabase
    .from('leads')
    .upsert({
      domain,
      summary,
      icp,
      raw_text: text,
      url,
      updated_at: new Date().toISOString()
    }, { onConflict: 'domain' });

  if (error) {
    console.error(`DB upsert failed for ${domain}:`, error.message);
    return null;
  }
  return data;
}

async function alreadyProcessed(domain) {
  const { data } = await supabase
    .from('leads')
    .select('domain')
    .eq('domain', domain)
    .maybeSingle();
  return !!data;
}

module.exports = { upsertLead, alreadyProcessed };

Why upsert? You'll re-run this pipeline as you add domains or want fresh data. The onConflict clause updates existing rows.

Step 5: Running the Full Pipeline

File: index.js

require('dotenv').config();
const fs = require('fs');
const { parse } = require('csv-parse/sync');
const { scrapeDomain } = require('./lib/scraper');
const { enrich } = require('./lib/enricher');
const { upsertLead, alreadyProcessed } = require('./lib/db');

async function main() {
  const csvPath = process.argv[2] || './domains.csv';
  const fileContent = fs.readFileSync(csvPath, 'utf-8');
  const records = parse(fileContent, {
    columns: true,
    skip_empty_lines: true
  });

  const domains = records.map(r => r.domain.trim().toLowerCase());
  console.log(`Loaded ${domains.length} domains`);

  for (const domain of domains) {
    console.log(`\nProcessing: ${domain}`);

    // Skip if already in DB
    if (await alreadyProcessed(domain)) {
      console.log(`  ✓ Already processed, skipping`);
      continue;
    }

    // Scrape
    const { text, url, error } = await scrapeDomain(domain);
    if (error) {
      console.log(`  ✗ Scrape error: ${error}`);
      continue;
    }
    console.log(`  Scraped ${text.length} chars`);

    // Enrich
    const enrichment = await enrich(domain, text);
    console.log(`  Summary: ${enrichment.summary?.slice(0, 100)}...`);
    console.log(`  ICP fit score: ${enrichment.icp?.fit_score}`);

    // Store
    await upsertLead({ domain, summary: enrichment.summary, icp: enrichment.icp, text, url });
    console.log(`  ✓ Stored in Supabase`);

    // Be gentle to free APIs
    await new Promise(resolve => setTimeout(resolve, 3000));
  }

  console.log('\nDone. Query your leads: supabase.io → SQL Editor → SELECT * FROM leads;');
}

main().catch(console.error);

Run it:

node index.js domains.csv

You'll see real-time progress. The 3-second delay respects OpenRouter's free-tier rate limits (~20 req/min). For larger lists, consider batching with async queues to control concurrency.

Sensible Extensions

1. Parallel scraping – Launch multiple Playwright contexts in a pool (e.g., p-queue with concurrency 3) to speed up scraping while staying polite.
2. Better text extraction – Use page.evaluate to grab specific selectors (e.g., document.querySelector('meta[name="description"]')?.content) for richer signals.
3. ICP scoring refinement – Send the prompt through a second model for validation, or use few-shot examples of known good/bad fits.
4. Webhook triggers – After upsert, POST to a Slack/Make webhook for real-time alerts on high-fit leads.
5. Scheduled runs – Wrap in a GitHub Action or cron job that runs daily against a Google Sheet of domains.
6. Export to CSV – Add a final step that writes leads table to a new CSV for your CRM.

Common Pitfalls

• OpenRouter 429s – Free models rate-limit aggressively. If you hit 429, increase the delay or rotate models (google/gemma-7b-it:free as fallback).
• Playwright timeouts – Some sites block headless browsers. Add headless: false for debugging, or set a custom userAgent and viewport. For persistent blocks, consider residential proxies.
• JSON parsing failures – Free models occasionally return malformed JSON. Add a retry with a stronger prompt, or fall back to regex extraction of key fields.
• Supabase connection – Ensure your anon key has insert/select permissions. In Supabase dashboard → Authentication → Policies, add a policy allowing inserts for the leads table if needed.
• Memory leaks – Always call browser.close() in a finally block. For very large lists, restart the browser every N domains.

FAQ

Q: Can I use this with a list of 10,000 domains? A: Yes, but it'll take hours on free-tier rate limits. Consider using OpenRouter's paid models (e.g., openai/gpt-3.5-turbo, $0.001/1K tokens) for speed, or parallelize scraping.

Q: What if a site is a single-page app? A: Playwright's domcontentloaded may be too fast. Switch to networkidle or add await page.waitForTimeout(3000) after navigation.

Q: How do I get the ICP score into my CRM? A: After the pipeline, export from Supabase as CSV, or use Supabase webhooks to push new rows to HubSpot/Salesforce.

Q: Is OpenRouter really free? A: Yes, for the listed models. You get free credits on signup, and free models don't consume credits—they're rate-limited instead. Perfect for batch enrichment.

Q: Can I run this in the cloud? A: Yes, deploy to a free-tier VM (Oracle Cloud, Google Cloud Run free tier) or even as a GitHub Action. Playwright runs fine in CI environments with playwright install --with-deps.