Build a Lead-Enrichment Agent That Scrapes Domains with Playwright & Gemini

What We're Building

A headless lead-enrichment agent. You drop in a CSV of company domains, the agent spins up a headless Chromium instance via Playwright, scrapes the homepage and core pages, extracts clean text, and ships it off to Google Gemini’s free tier. Gemini returns structured JSON with the company’s description, industry, estimated size, and key value props.

Core features:

• Batch domain ingestion from a plain CSV.
• Playwright-based intelligent scraping that handles SPAs, cookie banners, and redirects.
• Gemini Flash free-tier integration for structured extraction.
• Structured JSON output per domain, ready to pipe into a CRM.
• Respectful rate limiting so you don’t get IP-banned mid-run.

This isn’t a toy. It’s a production pattern you can bolt onto Clay, n8n, or a custom internal tool. By the end, you’ll have a single Python script that turns a list of URLs into a lead sheet you’d actually pay for.

Architecture Flow

Prerequisites (All Free Tier)

• Python 3.10+ – python.org/downloads
• Google Gemini API Key – Grab one from aistudio.google.com/apikey. The free tier gives you 15 requests per minute on Gemini 1.5 Flash, which is plenty for this pipeline.
• Playwright – Open-source browser automation. We’ll install it via pip and its bundled Chromium.

No credit card needed. No hidden quotas that’ll surprise you.

Step 1: Project Setup

mkdir lead-agent && cd lead-agent
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install playwright google-generativeai
playwright install chromium

Create a .env file:

GEMINI_API_KEY=your_key_here

Create domains.csv:

domain
stripe.com
vercel.com
linear.app

Step 2: Scraping with Playwright

We want a browser that loads the page, dismisses cookie banners, and waits for content to settle. Here’s the scraper module:

# scraper.py
from playwright.sync_api import sync_playwright
import time

def scrape_domain(domain: str) -> str:
    """Returns visible text content from a domain's homepage."""
    url = f"https://{domain}" if not domain.startswith("http") else domain

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        )
        page = context.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=15000)
            # Let JS-heavy pages settle
            page.wait_for_timeout(2000)

            # Aggressive cookie banner dismissal
            common_selectors = [
                "button:has-text('Accept')",
                "button:has-text('Accept All')",
                "button:has-text('OK')",
                ".cookie-accept",
                "#onetrust-accept-btn-handler"
            ]
            for selector in common_selectors:
                try:
                    page.click(selector, timeout=1000)
                    break
                except:
                    continue

            # Grab visible text from body, stripping scripts/styles
            text = page.inner_text("body")
            return text[:8000]  # Gemini Flash can handle this easily

        except Exception as e:
            return f"SCRAPE_ERROR: {str(e)}"
        finally:
            browser.close()

Why inner_text and not full HTML? LLMs choke on raw DOM. Visible text is higher signal, lower token count, and free-tier-friendly.

Step 3: Extracting Text Content

The scraper already returns clean-ish text. But you’ll hit pages with massive nav bars, footer links, and legal disclaimers that add noise. A production-grade version would strip boilerplate with trafilatura or readability-lxml, but for the free-tier pipeline, the 8000-character cap + Gemini’s native reasoning handles this well enough.

Step 4: Prompt Engineering for Gemini

This is where most builders phone it in. A lazy prompt gets hallucinated industries and made-up employee counts. A tight prompt with explicit output schema gets CRM-ready data.

# gemini_client.py
import google.generativeai as genai
import os
import json

API_KEY = os.getenv("GEMINI_API_KEY")
genai.configure(api_key=API_KEY)
model = genai.GenerativeModel("gemini-1.5-flash")

def extract_lead_profile(domain: str, scraped_text: str) -> dict:
    prompt = f"""You are a lead enrichment analyst. Given the scraped text from {domain}, extract a structured company profile.

Return ONLY valid JSON with these keys:
- company_name: string
- description: 2-sentence summary of what they do
- industry: string (pick from: Fintech, SaaS, E-commerce, Healthcare, AI/ML, DevTools, Cybersecurity, Other)
- estimated_size: string (pick from: 1-10, 11-50, 51-200, 201-1000, 1000+)
- key_products: list of strings (max 3)
- target_customer: string (who they sell to)
- confidence: string (High/Medium/Low based on text quality)

Scraped text:
{scraped_text[:6000]}"""

    try:
        response = model.generate_content(prompt)
        # Gemini sometimes wraps JSON in markdown fences
        raw = response.text.strip()
        if raw.startswith("```"):
            raw = raw.split("```")[1]
            if raw.startswith("json"):
                raw = raw[4:]
        return json.loads(raw)
    except Exception as e:
        return {"error": str(e), "domain": domain}

Why it works: We constrain the output schema explicitly. We cap text at 6000 characters to leave room for the prompt. We handle Gemini’s occasional markdown-wrapping quirk. The industry and size enums prevent free-form chaos.

Step 5: The Lead-Enrichment Pipeline

Now wire everything together with rate limiting and result persistence:

# main.py
import csv
import json
import time
import os
from dotenv import load_dotenv
from scraper import scrape_domain
from gemini_client import extract_lead_profile

load_dotenv()

def process_domains(csv_path="domains.csv", output_path="enriched_leads.json"):
    with open(csv_path, "r") as f:
        reader = csv.DictReader(f)
        domains = [row["domain"].strip() for row in reader if row.get("domain")]

    results = []
    for i, domain in enumerate(domains):
        print(f"[{i+1}/{len(domains)}] Processing {domain}...")

        # Scrape
        text = scrape_domain(domain)
        if text.startswith("SCRAPE_ERROR"):
            results.append({"domain": domain, "error": text})
            continue

        # Enrich
        profile = extract_lead_profile(domain, text)
        profile["domain"] = domain
        results.append(profile)

        # Respect Gemini free tier: 15 RPM = 1 request per 4 seconds
        time.sleep(5)

    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)

    print(f"Done. {len(results)} leads written to {output_path}")

if __name__ == "__main__":
    process_domains()

Step 6: Running the Agent

python main.py

You’ll see Playwright spin up headless Chromium, scrape each domain, and Gemini return structured profiles. Output lands in enriched_leads.json:

[
  {
    "domain": "stripe.com",
    "company_name": "Stripe",
    "description": "Stripe provides payment processing infrastructure for internet businesses. Their APIs enable companies to accept payments, manage subscriptions, and handle complex billing workflows.",
    "industry": "Fintech",
    "estimated_size": "1000+",
    "key_products": ["Stripe Payments", "Stripe Connect", "Stripe Billing"],
    "target_customer": "Online businesses, platforms, and marketplaces",
    "confidence": "High"
  }
]

Sensible Extensions

Multi-page scraping. The current version only hits the homepage. For richer profiles, follow the “About” and “Pricing” links. Use Playwright’s page.locator('a:has-text("About")').click() pattern, but add a depth limit (max 3 pages) to avoid spidering the entire site.

CRM webhook. Pipe results directly to HubSpot or Salesforce. Add a requests.post(WEBHOOK_URL, json=profile) call after each enrichment. Free-tier HubSpot forms accept JSON payloads with no authentication for testing.

Confidence-based filtering. Only push leads with confidence: "High" to your CRM. Flag Low results for manual review.

Parallelism. The free tier’s 15 RPM cap makes parallel scraping pointless for Gemini calls, but you can scrape domains concurrently with Playwright’s async API while queuing enrichment requests.

If you enjoy stitching agents like this into larger workflows, check out our guide on Build a YouTube-to-Blog Repurposing Agent with Gemini Free Tier and n8n — same pattern, different data source.

Common Pitfalls & Debugging

“SCRAPE_ERROR: net::ERR_NAME_NOT_RESOLVED” — The domain doesn’t resolve. Check for typos, add www. prefix handling, or skip and log.

Gemini returns ”error”: “Resource has been exhausted” — You hit the free-tier rate limit. Increase time.sleep() to 6-8 seconds or check your quota at aistudio.google.com.

JSON parse errors — Gemini occasionally returns malformed JSON. Add a retry loop with exponential backoff. In production, use a JSON repair library like json-repair.

Playwright timeout on SPAs — Some React apps take forever to hydrate. Increase wait_until="networkidle" and bump the timeout to 30 seconds for problematic domains.

Cookie banners still blocking content — The selector list covers 80% of banners. For stubborn ones, add domain-specific selectors or use Playwright’s page.evaluate() to remove overlay elements by z-index.

FAQ

Q: Is Gemini Flash actually free? Yes. Google’s free tier gives you 15 RPM and 1,500 requests per day on Gemini 1.5 Flash as of writing. That’s 1,500 lead profiles per day without spending a cent.

Q: Can I scrape 10,000 domains with this? You can, but you’ll need to batch across multiple days (free tier daily cap) or upgrade to pay-as-you-go. The architecture stays the same — just swap the API key.

Q: What if a site blocks Playwright? Rotate user agents, add --disable-blink-features=AutomationControlled to launch args, or route through residential proxies. For a deeper dive on agent architecture patterns, read Agentic Context Management: Treating Memory as an Architecture Problem.

Q: How does this compare to Clay or Apollo? Clay and Apollo are polished products with built-in waterfall enrichment. This agent is the free, hackable alternative. You control the logic, the prompts, and the output schema. When you need something custom — like extracting a very specific signal from a pricing page — this pattern wins.

Q: Can I deploy this as an API? Absolutely. Wrap process_domains in a FastAPI endpoint, add a background task queue, and you’ve got an internal enrichment microservice. If you’re building internal tools like this, you might also like Build a Daily Standup Bot That Collects Updates via DM and Posts a Slack Summary with Gemini.

Q: What’s the next skill to level up after this? Mastering structured extraction with LLMs is a core Forward Deployed Engineer skill. Once you’re comfortable with single-shot prompts like this, the next frontier is agentic loops where the model decides which pages to scrape and what questions to ask. FDE Coach has deep-dive resources on this exact progression — no bootcamp fluff, just patterns that ship.