Build a Screenshot-to-Code Agent Using Gemini Vision Free Tier & Playwright

What We’re Building & Feature List

We’re building a single-command CLI agent that eats a UI screenshot and spits out a standalone HTML/CSS file. No manual slicing, no guessing hex codes. You point the tool at a URL (or a local image), and it uses Google Gemini Vision’s free tier to generate production-ready markup.

Feature list

• --url flag: give it a live site and Playwright captures a full-page screenshot automatically.
• --image flag: feed it a local PNG/JPG screenshot.
• --output flag: destination path for the generated HTML file.
• Streaming progress so you aren’t staring at a blank terminal.
• Automatic retry with exponential backoff when the free-tier rate limit hits.
• Clean, self-contained HTML/CSS with no external dependencies (the model is prompted to inline everything).

By the end you’ll have a script you can drop into any design-to-code workflow—or chain into a larger agent that iterates on the output.

Architecture: How the Pieces Fit

The flow is intentionally linear. A screenshot (from Playwright or disk) is resized and converted to a base64 JPEG. That payload hits the Gemini Vision free tier with a structured prompt. The response is stripped of markdown fences, and the resulting HTML/CSS is written to disk. No orchestration framework, no vector DB—just a tight pipeline.

If you’ve built one of our other free-tier agents, like the YouTube-to-Blog Repurposer, you’ll recognize the pattern: capture media → call a free multimodal model → transform output. The same muscle memory applies.

Prerequisites (All Free-Tier)

Get your Gemini API key in 60 seconds:

1. Go to Google AI Studio.
2. Click “Create API key.”
3. Copy it and export it: export GEMINI_API_KEY="your-key-here"

The free tier gives you 1,500 requests per day on gemini-1.5-flash and gemini-1.5-pro. For screenshot-to-code, Flash is fast enough and keeps you well under the rate limit for iterative work.

Step 1: Project Scaffolding & Dependencies

Create a new directory and initialize:

mkdir screenshot-to-code-agent && cd screenshot-to-code-agent
npm init -y
npm install playwright @google/generative-ai sharp commander
npx playwright install chromium

What each package does:

• playwright: headless Chromium for capturing live-site screenshots.
• @google/generative-ai: official Google SDK for the Gemini API.
• sharp: fast image resizing and JPEG conversion (keeps base64 payloads small).
• commander: clean CLI argument parsing.

Your package.json needs "type": "module" because we’re using ES imports throughout.

Step 2: Capture a Screenshot with Playwright

Create src/capture.js:

import { chromium } from 'playwright';

export async function captureScreenshot(url, outputPath) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    viewport: { width: 1440, height: 900 },
    deviceScaleFactor: 2,
  });
  const page = await context.newPage();

  try {
    await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });
    // Let fonts and animations settle
    await page.waitForTimeout(1000);
    await page.screenshot({ path: outputPath, fullPage: true, type: 'png' });
    console.log(`✓ Screenshot saved to ${outputPath}`);
  } finally {
    await browser.close();
  }

  return outputPath;
}

Why deviceScaleFactor: 2: Gemini Vision works best with crisp input. 2x scaling gives the model more pixel detail without blowing up file size unreasonably.

Why fullPage: true: Most UIs scroll. You want the model to see the entire layout, not just the viewport.

Step 3: Call the Gemini Vision API (Free Tier)

Create src/gemini.js:

import { GoogleGenerativeAI } from '@google/generative-ai';
import sharp from 'sharp';
import fs from 'fs/promises';

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

const SYSTEM_PROMPT = `You are an expert frontend engineer. You receive a screenshot of a UI.
Return ONLY valid HTML with inline CSS. No markdown fences, no explanations.
Rules:
- Recreate the layout exactly as it appears.
- Use semantic HTML5 elements.
- Inline all CSS in a <style> tag in the <head>.
- Make it responsive (max-width container, relative units).
- Use system fonts or Google Fonts @import if the original uses a recognizable font.
- Output raw HTML starting with <!DOCTYPE html>.`;

export async function screenshotToCode(imagePath) {
  // Preprocess: resize if > 2048px wide, convert to JPEG, get base64
  const imageBuffer = await sharp(imagePath)
    .resize({ width: 2048, withoutEnlargement: true })
    .jpeg({ quality: 85 })
    .toBuffer();

  const base64Image = imageBuffer.toString('base64');

  const model = genAI.getGenerativeModel({
    model: 'gemini-1.5-flash',
    systemInstruction: SYSTEM_PROMPT,
  });

  const result = await model.generateContent([
    {
      inlineData: {
        mimeType: 'image/jpeg',
        data: base64Image,
      },
    },
    { text: 'Generate the HTML/CSS for this UI screenshot.' },
  ]);

  const response = result.response;
  const text = response.text();

  // Strip any lingering markdown fences
  return text
    .replace(/^```html?\s*/i, '')
    .replace(/```\s*$/, '')
    .trim();
}

Critical details:

• gemini-1.5-flash is the free-tier workhorse. It handles vision tasks well and has a generous 1,500 requests/day quota.
• sharp resizes to 2048px max width. Gemini’s vision models have no hard pixel limit on the free tier, but smaller payloads mean faster responses and fewer token costs.
• We strip markdown fences because the model sometimes wraps output in ```html despite explicit instructions not to.

Step 4: The Core Agent Loop

Create src/agent.js:

import { captureScreenshot } from './capture.js';
import { screenshotToCode } from './gemini.js';
import fs from 'fs/promises';
import path from 'path';
import { fileURLToPath } from 'url';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

async function runWithRetry(fn, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt === maxRetries) throw err;
      const waitMs = Math.pow(2, attempt) * 1000;
      console.log(`  ⚠ Retry ${attempt}/${maxRetries} in ${waitMs / 1000}s (${err.message})`);
      await new Promise((r) => setTimeout(r, waitMs));
    }
  }
}

export async function runAgent({ url, image, output }) {
  let screenshotPath;

  // Step 1: Get the screenshot
  if (url) {
    console.log(`📸 Capturing screenshot from ${url}...`);
    screenshotPath = path.join(__dirname, '..', 'temp_screenshot.png');
    await captureScreenshot(url, screenshotPath);
  } else if (image) {
    screenshotPath = image;
    console.log(`🖼 Using provided image: ${image}`);
  } else {
    throw new Error('Either --url or --image is required.');
  }

  // Step 2: Generate code with retry
  console.log('🤖 Sending to Gemini Vision...');
  const html = await runWithRetry(() => screenshotToCode(screenshotPath));

  // Step 3: Write output
  const outputPath = output || 'output.html';
  await fs.writeFile(outputPath, html, 'utf-8');
  console.log(`✅ HTML written to ${outputPath} (${Buffer.byteLength(html, 'utf-8')} bytes)`);

  // Cleanup temp screenshot if we captured it
  if (url && screenshotPath) {
    await fs.unlink(screenshotPath).catch(() => {});
  }

  return outputPath;
}

The retry logic handles free-tier rate limiting gracefully. Gemini’s free tier occasionally returns 429s under bursty load; exponential backoff with a 2^n multiplier clears most transient failures.

Step 5: Saving & Previewing the Output

The agent already writes the file in Step 4. Let’s add a quick preview helper so you can iterate faster. Create src/preview.js:

import { chromium } from 'playwright';
import path from 'path';

export async function previewHtml(htmlPath) {
  const browser = await chromium.launch({ headless: false });
  const page = await browser.newPage();
  const absolutePath = path.resolve(htmlPath);
  await page.goto(`file://${absolutePath}`);
  console.log('🌐 Preview opened in Chromium. Close the browser to exit.');
  // Keep alive until user closes the browser
  await new Promise(() => {});
}

This isn’t wired into the main CLI by default—it’s a utility you can call when you want a side-by-side comparison.

How to Run the CLI

Wire everything together in index.js:

#!/usr/bin/env node
import { Command } from 'commander';
import { runAgent } from './src/agent.js';

const program = new Command();

program
  .name('screenshot-to-code')
  .description('Generate HTML/CSS from a UI screenshot using Gemini Vision free tier')
  .option('-u, --url <url>', 'URL of the site to screenshot')
  .option('-i, --image <path>', 'Path to a local screenshot image')
  .option('-o, --output <path>', 'Output HTML file path', 'output.html')
  .action(async (options) => {
    if (!options.url && !options.image) {
      console.error('Error: You must provide either --url or --image.');
      process.exit(1);
    }
    await runAgent(options);
  });

program.parse();

Make it executable and run:

chmod +x index.js

# From a live URL
GEMINI_API_KEY="your-key" node index.js --url "https://example.com" --output landing.html

# From a local screenshot
GEMINI_API_KEY="your-key" node index.js --image ./mockup.png --output mockup.html

Expected output:

📸 Capturing screenshot from https://example.com...
✓ Screenshot saved to temp_screenshot.png
🤖 Sending to Gemini Vision...
✅ HTML written to landing.html (4823 bytes)

Open landing.html in a browser. The generated page will be a close approximation—not pixel-perfect, but structurally sound and immediately usable as a starting point.

Sensible Extensions

Once the core loop works, you can bolt on capabilities that make this genuinely useful in a daily workflow:

1. Iterative refinement loop: Pipe the generated HTML back through Playwright, screenshot that, and ask Gemini to diff and fix layout discrepancies. Two passes often eliminate 80% of initial errors.
2. Component extraction mode: Add a --component flag that prompts the model to output a single React/Vue/Svelte component instead of a full page. This is where the agent starts replacing manual slice-and-dice work.
3. Batch processing: Feed it a directory of mockups and generate a corresponding component library. Combine with a simple glob and Promise.all (respecting rate limits).
4. Design-token extraction: Ask the model to also output a JSON block with the color palette, font stack, and spacing scale it inferred. This is gold for maintaining consistency across generated pages.
5. Diff mode: Compare the generated page screenshot against the original and produce a visual diff heatmap. Playwright’s page.screenshot() with clip makes region-specific comparison straightforward.

If you enjoy wiring free models into practical automation, the Job Application Autofill Agent follows a similar pattern—browser automation meets LLM reasoning—and is a natural next build.

Common Pitfalls

FAQ

Q: How accurate is the generated code? Expect 70-90% structural accuracy on the first pass. Layout containers, flex/grid choices, and color values are usually correct. Fine spacing, exact font sizes, and subtle shadows may drift. The iterative refinement extension closes most of that gap.

Q: Can I use this for production code? Not directly. Treat the output as a high-fidelity scaffold. You’ll want to review accessibility, add interactivity, and refactor into your component framework. It’s a massive accelerator, not a replacement for engineering judgment.

Q: Does this work with dark-mode screenshots? Yes. The model handles both light and dark UIs without special prompting. If you want to force a specific mode, capture the screenshot accordingly.

Q: What’s the cost if I exceed the free tier? Gemini 1.5 Flash is pay-as-you-go beyond the free quota at roughly $0.00002 per image and $0.0000025 per 1k characters of output. For context, 1,000 screenshots would cost approximately $0.02 in image tokens plus a few cents for output tokens. It’s effectively free for individual use.

Q: Can I swap in a different vision model? Absolutely. The architecture is model-agnostic. Swap @google/generative-ai for the OpenAI SDK (GPT-4o), Anthropic (Claude 3.5 Sonnet), or a local model via Ollama. The preprocessing and post-processing layers stay identical.

Q: Why not use a specialized screenshot-to-code tool? Because those tools lock you into their ecosystem and pricing. Building it yourself with free-tier LLMs gives you full control over the prompt, the output format, and the iteration loop. Plus, the skill of wiring vision models into automation transfers to dozens of other use cases—like the WhatsApp Customer Support Agent we built with n8n and Qdrant.

Q: How do I handle mobile screenshots? Pass a --viewport flag to Playwright with mobile dimensions (e.g., 390x844) or feed in a mobile screenshot directly. The model adapts its output to the viewport it sees—no code changes needed.

If you want to go deeper on the skills that make builds like this second nature, we’ve written about the highest-leverage FDE skills in the AI era—pattern recognition, model wrangling, and shipping velocity matter more than any single framework.