Turn UI Screenshots Into Production Code With a Free Vision LLM

What We're Building

We're building a single Python script that does one thing exceptionally well: it takes a URL, captures a full-page screenshot with Playwright, sends that image to a free Hugging Face vision model (Llava-v1.6), and receives clean, usable HTML/CSS or React component code in return. No API keys that expire. No credit card required. Just raw engineering leverage.

Feature list:

• Captures full-page or viewport-specific screenshots via Playwright
• Calls Hugging Face's free Inference API with a vision-language model
• Returns structured HTML/CSS or React/JSX code from a single prompt
• Handles rate limits gracefully with retry logic
• Saves both the screenshot and generated code to disk for immediate use

If you've ever stared at a Figma mockup and wished you could just generate the first draft, this pipeline is your new best friend. It's also a perfect starting point for building a competitor monitoring agent that alerts on site changes — swap the prompt and you're diffing production UIs against design files automatically.

Architecture Overview

Here's how the pieces fit together:

The flow is intentionally simple: capture, encode, prompt, receive, save. No orchestration layer, no queue, no state management. A single Python file you can run from the terminal. The magic sits entirely in the prompt engineering — we're asking a general-purpose vision model to act as a specialized UI-to-code translator.

Prerequisites

Everything here is free-tier or open-source. Here's exactly what you need:

After installing Playwright, run playwright install chromium to pull the browser binary. For Hugging Face, create a free account, go to Settings → Access Tokens, and generate a read token. The free tier gives you rate-limited access to thousands of models, including the vision models we need.

Step 1: Capturing High-Quality Screenshots

Playwright is the gold standard for browser automation. We'll use it to load a page and capture a full-height screenshot. The key detail: we want full_page=True so we capture the entire scrollable page, not just the viewport.

from playwright.sync_api import sync_playwright

def capture_screenshot(url: str, output_path: str = "screenshot.png") -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(viewport={"width": 1440, "height": 900})
        page.goto(url, wait_until="networkidle")
        page.screenshot(path=output_path, full_page=True)
        browser.close()
    return output_path

wait_until="networkidle" is critical — it waits until there are no more than 2 network connections for at least 500ms. This catches lazy-loaded images and async-rendered components. Without it, you'll screenshot half-loaded pages and the model will generate code for a broken UI.

Step 2: Setting Up the Hugging Face Inference Client

We're using the free Inference API, not a paid endpoint. The model we'll target is llava-hf/llava-1.5-7b-hf, a 7B parameter vision-language model that runs on Hugging Face's shared infrastructure. It's not the newest model on the block, but it's free and surprisingly capable for UI-to-code tasks.

import base64
import requests
import time
import os

HF_TOKEN = os.environ.get("HF_TOKEN", "your-token-here")
API_URL = "https://api-inference.huggingface.co/models/llava-hf/llava-1.5-7b-hf"
HEADERS = {"Authorization": f"Bearer {HF_TOKEN}"}

def query_vision_model(image_path: str, prompt: str, max_retries: int = 3) -> str:
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    
    payload = {
        "inputs": prompt,
        "parameters": {"max_new_tokens": 2048, "temperature": 0.2}
    }
    
    for attempt in range(max_retries):
        response = requests.post(API_URL, headers=HEADERS, json=payload, data=image_bytes)
        
        if response.status_code == 200:
            result = response.json()
            return result[0]["generated_text"] if isinstance(result, list) else result["generated_text"]
        
        if response.status_code == 503:
            # Model is loading — wait and retry
            wait_time = (attempt + 1) * 10
            print(f"Model loading, retrying in {wait_time}s...")
            time.sleep(wait_time)
            continue
        
        raise Exception(f"API error {response.status_code}: {response.text}")
    
    raise Exception("Max retries exceeded")

Three things to note: we send the image as raw bytes in the request body (not base64-encoded in JSON — that's a common mistake), we set temperature=0.2 for deterministic code output, and we handle 503 errors because free-tier models spin down when idle and need 30-60 seconds to warm up.

Step 3: Crafting the Vision Prompt for Code Generation

The prompt is where you win or lose. A vague "generate code for this" will give you a paragraph describing the UI. We need the model to output only code, in a specific format. Here's the prompt template that works:

CODE_PROMPT = """You are an expert frontend developer. Analyze this UI screenshot and generate the complete HTML and CSS code to recreate it.

Requirements:
- Output ONLY valid HTML with inline or embedded CSS. No explanations, no markdown fences.
- Use semantic HTML5 elements (header, nav, main, section, footer).
- Match the layout, colors, typography, and spacing as closely as possible.
- Include placeholder images using https://placehold.co/400x300 URLs.
- Make it responsive with a max-width container and mobile-friendly breakpoints.

Start your response with <!DOCTYPE html> and end with </html>."""

For React output, swap the last line with:

REACT_PROMPT = """...same as above...
- Output a single React functional component using TypeScript.
- Use inline styles or a CSS-in-JS approach (no external CSS files).
- Export the component as default.

Start your response with 'import React from "react";' and output only the component code."""

The instruction to output only code and start with a specific token is a prompting pattern that dramatically reduces the model's tendency to add conversational fluff. If you still get markdown fences, we'll strip them in post-processing.

Step 4: The Complete Python Script

Here's the full script. Save it as screenshot_to_code.py:

#!/usr/bin/env python3
"""Turn UI screenshots into production code using free Hugging Face vision models."""

import argparse
import base64
import os
import re
import time
import requests
from playwright.sync_api import sync_playwright

HF_TOKEN = os.environ.get("HF_TOKEN")
if not HF_TOKEN:
    raise RuntimeError("Set HF_TOKEN environment variable with your Hugging Face token")

API_URL = "https://api-inference.huggingface.co/models/llava-hf/llava-1.5-7b-hf"
HEADERS = {"Authorization": f"Bearer {HF_TOKEN}"}

HTML_PROMPT = """You are an expert frontend developer. Analyze this UI screenshot and generate complete HTML and CSS code.

Output ONLY valid HTML with embedded CSS in a <style> tag. No markdown fences, no explanations.
Use semantic HTML5 elements. Match layout, colors, typography, and spacing exactly.
Include placeholder images with https://placehold.co/400x300.
Make it responsive with max-width container and mobile breakpoints.
Start with <!DOCTYPE html> and end with </html>."""

REACT_PROMPT = """You are an expert React developer. Analyze this UI screenshot and generate a complete React component.

Output ONLY a TypeScript React functional component. No markdown fences, no explanations.
Use inline styles. Export as default. Match layout, colors, typography, and spacing exactly.
Include placeholder images with https://placehold.co/400x300.
Start with 'import React from "react";'."""


def capture_screenshot(url: str, output_path: str = "screenshot.png") -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(viewport={"width": 1440, "height": 900})
        page.goto(url, wait_until="networkidle", timeout=30000)
        page.screenshot(path=output_path, full_page=True)
        browser.close()
    print(f"Screenshot saved: {output_path}")
    return output_path


def query_model(image_path: str, prompt: str, max_retries: int = 5) -> str:
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    
    payload = {"parameters": {"max_new_tokens": 2048, "temperature": 0.2}}
    
    for attempt in range(max_retries):
        response = requests.post(
            API_URL,
            headers=HEADERS,
            json=payload,
            data=image_bytes,
            timeout=120
        )
        
        if response.status_code == 200:
            result = response.json()
            text = result[0]["generated_text"] if isinstance(result, list) else result["generated_text"]
            # Strip the prompt echo if present
            if text.startswith(prompt):
                text = text[len(prompt):].strip()
            return text
        
        if response.status_code == 503:
            wait = min((attempt + 1) * 15, 90)
            print(f"Model loading (attempt {attempt + 1}/{max_retries}), waiting {wait}s...")
            time.sleep(wait)
            continue
        
        response.raise_for_status()
    
    raise RuntimeError("Model failed to respond after maximum retries")


def clean_output(raw: str, output_type: str) -> str:
    # Remove markdown code fences if present
    cleaned = re.sub(r"^```[\w]*\n?", "", raw.strip())
    cleaned = re.sub(r"\n?```$", "", cleaned)
    
    if output_type == "react":
        if not cleaned.startswith("import"):
            cleaned = 'import React from "react";\n\n' + cleaned
    elif output_type == "html":
        if not cleaned.startswith("<!DOCTYPE"):
            cleaned = "<!DOCTYPE html>\n<html lang=\"en\">\n<head><meta charset=\"UTF-8\"></head>\n<body>\n" + cleaned + "\n</body>\n</html>"
    
    return cleaned


def main():
    parser = argparse.ArgumentParser(description="Convert UI screenshots to code")
    parser.add_argument("url", help="URL of the page to screenshot and convert")
    parser.add_argument("--output", "-o", default="output", help="Output directory")
    parser.add_argument("--type", "-t", choices=["html", "react"], default="html", help="Output code type")
    args = parser.parse_args()
    
    os.makedirs(args.output, exist_ok=True)
    
    screenshot_path = os.path.join(args.output, "screenshot.png")
    capture_screenshot(args.url, screenshot_path)
    
    prompt = REACT_PROMPT if args.type == "react" else HTML_PROMPT
    print(f"Querying model for {args.type} code...")
    raw_code = query_model(screenshot_path, prompt)
    
    cleaned = clean_output(raw_code, args.type)
    
    ext = ".tsx" if args.type == "react" else ".html"
    output_path = os.path.join(args.output, f"output{ext}")
    with open(output_path, "w") as f:
        f.write(cleaned)
    
    print(f"Generated {args.type} code saved to: {output_path}")
    print(f"Preview: {cleaned[:200]}...")


if __name__ == "__main__":
    main()

Step 5: Running It End-to-End

Set your token and run:

export HF_TOKEN="hf_yourReadTokenHere"
python screenshot_to_code.py https://example.com --output ./example-output --type html

Expected output:

Screenshot saved: ./example-output/screenshot.png
Querying model for html code...
Model loading (attempt 1/5), waiting 15s...
Generated html code saved to: ./example-output/output.html

First run almost always hits the 503 because the model needs to load. Subsequent runs are faster. The generated HTML won't be pixel-perfect — this is a 7B model, not GPT-4V — but it'll give you a solid structure, correct color palette, and roughly correct layout. It's an 80/20 tool: 80% of the work done in 20 seconds, leaving you to fine-tune the details.

Extending the Pipeline

Once you have the basic flow working, here's where to take it:

Multi-page scraping: Loop over a list of URLs and generate a component library from an entire site. This pairs naturally with the YouTube-to-blog repurposing agent pattern — same idea, different input modality.

Diff-based refactoring: Capture a screenshot before and after a design change, send both to the model, and ask it to generate only the CSS diff. Hugely useful when you're iterating on a design system.

Automated visual regression: Hook this into CI. On every PR, screenshot the affected pages, generate code, and compare the DOM structure against the existing implementation. Catch visual regressions before they hit production. This is a natural extension of the competitor monitoring agent architecture.

Batch processing with a queue: Wrap the script in a simple Redis queue to handle dozens of URLs without hitting rate limits. The free tier allows ~30 requests per hour — respect it.

Common Pitfalls & Fixes

Model returns a description instead of code: Your prompt isn't directive enough. Add "Output ONLY code" and specify the exact starting token. If the model still wanders, lower the temperature further (0.1) and reduce max_new_tokens to 1024 to force conciseness.

503 errors on every request: The free tier unloads models after ~15 minutes of inactivity. The retry logic handles this, but if you're getting consistent 503s, check that your token has Inference API access enabled (it's a checkbox in your HF settings).

Screenshots are blank or incomplete: Some sites aggressively lazy-load or use client-side rendering that networkidle doesn't catch. Add page.wait_for_timeout(3000) after navigation to let animations settle. For SPAs, use page.wait_for_selector("body") with a specific element you know renders last.

Generated code has broken image URLs: The model sometimes hallucinates image sources. The prompt explicitly asks for placehold.co URLs, but if it still generates random ones, add a post-processing step that regex-replaces src="http..." with placeholder URLs.

Token usage is high and you're getting truncated output: Full-page screenshots of long pages produce massive image payloads. The model has a context window limit. For very long pages, capture viewport-only screenshots (full_page=False) or resize images before sending.

FAQ

Q: Why Llava-v1.6 instead of a newer model? A: It's free, always available on the Inference API, and has strong vision-language performance for its size. Newer models like Llama 3.2 Vision exist but may not be on the free tier yet. The architecture is model-agnostic — swap the API_URL to try others.

Q: Can I use this for production code generation? A: Not directly. The output is a strong first draft — use it to skip the blank-canvas problem, then refine manually. For production pipelines, you'd want a larger model, fine-tuning on your component library, and a human-in-the-loop review step.

Q: How does this compare to paid tools like v0 or screenshot-to-code SaaS? A: Those tools use larger models and are more polished, but they cost money and send your screenshots to third-party servers. This script runs locally (except the model inference) and costs nothing. It's the difference between a managed service and owning the pipeline.

Q: What if I want to generate React with Tailwind instead of inline styles? A: Modify the prompt. Replace "Use inline styles" with "Use Tailwind CSS classes (assume a Tailwind CDN is available)". The model understands Tailwind well enough to generate reasonable class strings.

Q: Can I run the model locally instead of using the API? A: Yes, if you have a GPU with 8GB+ VRAM. Use the transformers library to load llava-hf/llava-1.5-7b-hf locally. This removes rate limits entirely. The tradeoff is hardware cost versus convenience.

---

This pipeline is a concrete example of what a Forward Deployed Engineer builds daily: taking existing, free building blocks and wiring them together to solve a real problem. If you want to develop the instincts to spot these opportunities and build them fast, FDE Coach is where you sharpen that skillset.