Build a Lead-Enrichment Agent with Playwright and Gemini (Free Tier)

What We're Building

A Python agent that takes a list of company domains, visits each one with a headless browser, extracts the visible text, and feeds it to Gemini Flash to pull out structured firmographic data—industry, employee count, key contacts, and funding stage. The output is a clean CSV you can pipe directly into your CRM or outbound workflow.

Feature list:

• Headless browsing with Playwright (bypasses JS-only sites that requests can't touch)
• Intelligent text extraction: grabs the hero, about page, and careers page
• Gemini Flash 1.5 free tier for structured JSON extraction
• CSV output with domain, company name, industry, size bucket, contacts, and confidence score
• Rate limiting and retry logic baked in (free tier quotas are tight)
• Configurable via a simple YAML config file

Architecture Overview

The flow is linear but fault-tolerant. If Playwright times out on a domain, we log the failure and move on. If Gemini returns malformed JSON, we retry with a stricter prompt. Every enriched record lands in the output CSV immediately, so a crash at row 200 doesn't lose the first 199.

Prerequisites (All Free Tier)

• Python 3.10+ – python.org/downloads
• Playwright – pip install playwright then playwright install chromium. Open-source, no API key needed.
• Google Gemini API key – Free tier gives 15 requests per minute, 1,500 per day. Grab a key at aistudio.google.com/apikey. No credit card required for the free tier.
• python-dotenv – Keeps your API key out of source. pip install python-dotenv

Create a .env file:

GEMINI_API_KEY=your-key-here

Step 1: Project Setup and Dependencies

mkdir lead-enrichment-agent && cd lead-enrichment-agent
python -m venv venv && source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install playwright python-dotenv google-generativeai pydantic
playwright install chromium

Create config.yaml:

max_pages_per_domain: 3
timeout_ms: 15000
rate_limit_rpm: 10  # stay under Gemini's 15 RPM free limit
model: "gemini-1.5-flash"
output_csv: "enriched_leads.csv"

Create your input file domains.csv:

domain
stripe.com
linear.app
vercel.com

Step 2: Scraping Website Text with Playwright

We're not screenshotting—we're pulling innerText from key pages. Playwright handles SPAs, cookie banners, and redirect chains that break lighter tools.

# scraper.py
import asyncio
from playwright.async_api import async_playwright

async def scrape_domain(browser, domain: str, timeout_ms: int = 15000) -> str:
    pages_to_visit = [
        f"https://{domain}",
        f"https://{domain}/about",
        f"https://{domain}/careers"
    ]
    collected_text = []

    for url in pages_to_visit:
        try:
            page = await browser.new_page()
            await page.goto(url, timeout=timeout_ms, wait_until="domcontentloaded")
            # Dismiss common cookie banners if present
            await page.evaluate("""
                () => {
                    const buttons = document.querySelectorAll('button');
                    buttons.forEach(b => {
                        if (/accept|agree|ok|got it/i.test(b.innerText)) b.click();
                    });
                }
            """)
            await page.wait_for_timeout(1000)
            text = await page.inner_text("body")
            collected_text.append(f"--- {url} ---\n{text[:3000]}")  # truncate per page
            await page.close()
        except Exception as e:
            print(f"  [!] Failed {url}: {e}")

    return "\n\n".join(collected_text)

The cookie-banner dismiss is a pragmatic hack. It won't catch every variant, but it handles 80% of GDPR walls without adding a heavyweight consent-management library. We truncate each page to 3,000 characters because Gemini's free tier has a 32k token context window and we want room for the prompt.

Step 3: Structuring Data with Gemini Flash

Gemini Flash is fast, cheap (free), and supports constrained JSON output via response schemas. We'll define exactly what fields we want using Pydantic, then pass the schema to the API.

# extractor.py
import google.generativeai as genai
import json
import pydantic
from typing import Optional

class CompanyProfile(pydantic.BaseModel):
    company_name: str
    industry: str
    employee_count_bucket: str  # "1-10", "11-50", "51-200", "201-1000", "1000+"
    key_contacts: list[str]
    funding_stage: str  # "bootstrapped", "seed", "series-a", "series-b+", "public", "unknown"
    confidence_score: float  # 0.0 to 1.0

def build_prompt(domain: str, scraped_text: str) -> str:
    return f"""You are a lead enrichment analyst. Given the scraped website text for {domain}, extract a structured company profile.

Rules:
- Only use information explicitly present in the text. Do not hallucinate.
- If a field cannot be determined, use "unknown" for strings and 0.0 for confidence.
- For employee_count_bucket, infer from team page mentions, LinkedIn blurbs, or About page language.
- For key_contacts, extract full names and titles of executives or founders mentioned.
- Set confidence_score based on how much evidence supports the extraction (0.0 = nothing found, 1.0 = explicitly stated).

Website text:
{scraped_text[:25000]}"""

def extract_profile(domain: str, scraped_text: str, api_key: str, model_name: str) -> dict:
    genai.configure(api_key=api_key)
    model = genai.GenerativeModel(model_name)

    response = model.generate_content(
        build_prompt(domain, scraped_text),
        generation_config=genai.GenerationConfig(
            response_mime_type="application/json",
            response_schema=CompanyProfile,
            temperature=0.1  # low temp for factual extraction
        )
    )

    return json.loads(response.text)

The response_schema parameter is the killer feature here. It forces Gemini to output valid JSON matching our Pydantic model every time. No regex parsing, no "I apologize, but as an AI..." in the middle of your data pipeline.

Step 4: Orchestrating the Full Enrichment Pipeline

This is where we wire everything together with rate limiting, error handling, and incremental CSV writes.

# main.py
import asyncio
import csv
import os
import time
import yaml
from dotenv import load_dotenv
from scraper import scrape_domain
from extractor import extract_profile
from playwright.async_api import async_playwright

load_dotenv()

async def main():
    with open("config.yaml") as f:
        config = yaml.safe_load(f)

    with open("domains.csv") as f:
        domains = [row["domain"].strip() for row in csv.DictReader(f)]

    output_path = config["output_csv"]
    fieldnames = ["domain", "company_name", "industry", "employee_count_bucket",
                  "key_contacts", "funding_stage", "confidence_score"]

    # Write header if file doesn't exist
    if not os.path.exists(output_path):
        with open(output_path, "w", newline="") as f:
            writer = csv.DictWriter(f, fieldnames=fieldnames)
            writer.writeheader()

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        requests_this_minute = 0
        minute_start = time.time()

        for domain in domains:
            print(f"\n[+] Processing {domain}...")

            # Rate limit: reset counter every 60 seconds
            if time.time() - minute_start >= 60:
                requests_this_minute = 0
                minute_start = time.time()

            if requests_this_minute >= config["rate_limit_rpm"]:
                wait_time = 60 - (time.time() - minute_start)
                print(f"  [~] Rate limit approaching, waiting {wait_time:.0f}s...")
                await asyncio.sleep(wait_time)
                requests_this_minute = 0
                minute_start = time.time()

            # Scrape
            scraped = await scrape_domain(browser, domain, config["timeout_ms"])
            if not scraped.strip():
                print(f"  [-] No text extracted for {domain}, skipping.")
                continue

            # Extract
            try:
                profile = extract_profile(
                    domain,
                    scraped,
                    os.getenv("GEMINI_API_KEY"),
                    config["model"]
                )
                requests_this_minute += 1
            except Exception as e:
                print(f"  [!] Gemini extraction failed: {e}")
                continue

            # Write row immediately
            row = {"domain": domain, **profile}
            with open(output_path, "a", newline="") as f:
                writer = csv.DictWriter(f, fieldnames=fieldnames)
                writer.writerow(row)

            print(f"  [✓] {profile.get('company_name', 'Unknown')} | "
                  f"{profile.get('industry', '?')} | "
                  f"{profile.get('employee_count_bucket', '?')}")

        await browser.close()

    print(f"\nDone. Results in {output_path}")

if __name__ == "__main__":
    asyncio.run(main())

Step 5: Running the Agent

python main.py

Expected output:

[+] Processing stripe.com...
  [✓] Stripe | Financial Technology | 1000+
[+] Processing linear.app...
  [✓] Linear | Project Management Software | 51-200
[+] Processing vercel.com...
  [✓] Vercel | Cloud Platform | 201-1000

Done. Results in enriched_leads.csv

The CSV will have one row per domain with all extracted fields. Open it in Excel, Google Sheets, or pipe it into your next script.

Sensible Extensions

Once the core pipeline works, here's where you take it next:

1. Add LinkedIn enrichment. After scraping the website, hit the LinkedIn company page (publicly accessible without auth for basic info) and merge the data. This dramatically improves employee count accuracy.

2. Parallelize with a semaphore. Right now we process sequentially. Add asyncio.Semaphore(3) to run three Playwright browsers concurrently while staying under Gemini's rate limit. This cuts runtime by ~60%.

3. Store raw scrapes. Save the raw text to a SQLite database before extraction. If you tweak your extraction prompt later, you can re-process without re-scraping and hammering target sites.

4. Add a confidence threshold filter. Only write rows where confidence_score > 0.5 to the output CSV. Flag low-confidence domains for manual review.

5. Deploy as a scheduled job. Wrap it in a cron job or GitHub Action that runs weekly against your CRM's new leads. The architecture we built at Build a GitHub PR Review Bot that Comments on Code with Groq's Free API follows the same "free-tier cron agent" pattern you can adapt here.

Common Pitfalls

• Gemini quota exhaustion. The free tier gives 1,500 requests/day. If you're enriching 2,000 domains, you'll hit the wall. Either batch across multiple days or upgrade to pay-as-you-go ($0.075 per 1k requests for Flash).

• Playwright timeouts on slow sites. Some company websites are built on bloated WordPress themes that take 30+ seconds to load. The timeout_ms in config defaults to 15 seconds. Increase it if you're getting empty scrapes.

• JavaScript-rendered content. Playwright handles SPAs, but some sites lazy-load content on scroll. You may need to add await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") before extracting text.

• Gemini hallucinating contacts. The model sometimes invents names that sound plausible but aren't on the page. The confidence_score field helps you filter these, but for production use cases, cross-reference with LinkedIn before sending outreach.

• Blocked by bot detection. Some sites (looking at you, Cloudflare-protected ones) will serve a CAPTCHA. Playwright with the default Chromium profile gets through most, but you may need to add a user_agent override or --disable-blink-features=AutomationControlled launch arg for stubborn targets.

FAQ

Q: Why Playwright instead of requests + BeautifulSoup? A: Roughly 40% of company websites are SPAs that return an empty <body> without JavaScript execution. Playwright renders the DOM like a real browser. If you've ever tried to scrape a React site with requests, you know the pain. This is the same approach we use in Build a Screenshot-to-React Agent with Google Gemini Flash and Free Hosting for capturing rendered pages.

Q: Can I use a different LLM? A: Yes. Swap the extract_profile function to call Groq (Llama 3.1 8B is free) or OpenAI. The Pydantic schema approach works with any provider that supports structured output. Groq's free tier is faster but less accurate on nuanced extraction tasks like inferring funding stage from vague About page language.

Q: How do I handle non-English websites? A: Add "Translate all extracted fields to English" to the prompt. Gemini handles 100+ languages natively. The key_contacts field will preserve original names.

Q: What if a domain redirects to a different URL? A: Playwright follows redirects by default. The scraped text will come from the final destination. If you want to log redirects, check page.url after goto().

Q: Is this production-ready? A: For internal lead enrichment on a few hundred domains per day, absolutely. For customer-facing production pipelines, you'll want persistent browser contexts, a proper job queue (Redis), and exponential backoff on retries. That handoff from prototype to production system is exactly the pattern covered in Scaling Yourself: When an FDE Hands Off to Core Engineering for Productionization.