Convert Screenshots to Frontend Code Using Gemini 1.5 Flash Free Vision Model

What We're Building

We're building a zero-cost command-line utility that ingests a UI screenshot and outputs a functional, single-file HTML/CSS/JS replica. No API keys that require billing, no cloud services that expire after a trial. Just a Python script, Google's Gemini 1.5 Flash free tier, and Playwright for visual validation.

Feature list:

• Accepts any PNG/JPG screenshot via command-line argument
• Encodes the image as base64 and ships it to Gemini's vision endpoint
• Prompts the model to extract layout, colors, fonts, and interactive elements
• Generates a standalone .html file that renders in any browser
• Validates the output by taking a Playwright screenshot of the generated page
• Entirely free-tier: no credit card required for the Gemini API key

This is a practical tool for rapid prototyping, design handoff, or reverse-engineering a layout you saw online. It's not going to produce production-grade React components, but it will get you 80% of the way there in 30 seconds.

Architecture: The Pixel-to-Code Pipeline

Before we write a line of code, let's map the data flow. The system has four stages: input encoding, model inference, output extraction, and visual validation.

The critical design decision: we ask Gemini to wrap its generated code in a markdown code fence (```html ... ```). This makes parsing deterministic. Without this constraint, the model might return explanatory text mixed with code, and regex extraction becomes fragile.

Prerequisites: Free Tools You'll Need

Everything here operates within free tiers. No credit card charges will appear.

Get your API key: Go to Google AI Studio, click "Get API Key", and create one. The free tier gives you 1,500 requests per day on Gemini 1.5 Flash. That's enough to convert 1,500 screenshots daily without spending a cent.

Step 1: Setting Up the Python Environment

Create a project directory and install dependencies:

mkdir screenshot-to-code && cd screenshot-to-code
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install google-generativeai playwright
playwright install chromium

Store your API key as an environment variable. Never hardcode secrets:

export GEMINI_API_KEY="your-key-here"

Now create convert.py and start with the imports:

import os
import sys
import base64
import re
import argparse
from pathlib import Path

import google.generativeai as genai
from playwright.sync_api import sync_playwright

Step 2: Encoding the Screenshot for Gemini

Gemini's vision endpoint accepts images as base64-encoded strings or as file uploads. Base64 is simpler for a CLI tool — no intermediate upload step, no temporary URIs.

def encode_image_to_base64(image_path: str) -> tuple[str, str]:
    """Read an image file and return (base64_string, mime_type)."""
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Screenshot not found: {image_path}")

    suffix = path.suffix.lower()
    mime_map = {
        ".png": "image/png",
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".webp": "image/webp",
    }
    mime_type = mime_map.get(suffix, "image/png")

    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8"), mime_type

This function handles the three formats you'll realistically encounter: PNG, JPEG, and WebP. The MIME type matters — Gemini uses it to decode the image correctly.

Step 3: Crafting the Prompt for Pixel-Perfect Extraction

Prompt engineering is the entire game here. A vague prompt produces vague HTML. We need to constrain the output format and give explicit instructions about what to extract.

SYSTEM_PROMPT = """You are a frontend engineer converting UI screenshots into functional HTML/CSS/JS.

Analyze the provided screenshot and generate a SINGLE, complete HTML file that replicates the design as closely as possible.

Rules:
1. Output ONLY a markdown code block containing the full HTML: ```html ... ```
2. Include all CSS inline within a <style> tag in the <head>.
3. Include all JavaScript inline within a <script> tag at the end of <body>.
4. For images/icons you cannot reproduce, use inline SVG placeholders or emoji.
5. Match colors, fonts, spacing, and border radii as precisely as you can.
6. Use system font stacks: `font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;`
7. Make buttons and links look interactive (hover states via CSS).
8. Do NOT include any explanation, summary, or commentary outside the code block.
9. Do NOT use external CDN links, images, or fonts. Everything must be self-contained.
"""

Why these constraints? Rule 9 prevents the model from pulling in Bootstrap or Tailwind CDNs, which would make the output dependent on external resources. Rule 4 forces the model to generate placeholder assets rather than hallucinating broken <img src="..."> tags. Rule 1 makes parsing trivial.

Step 4: Parsing the LLM Response and Writing the HTML File

With the prompt enforcing a code fence, extraction is a simple regex:

def extract_html_from_response(text: str) -> str:
    """Extract HTML content from a markdown code block."""
    # Match ```html ... ``` or ```HTML ... ```
    pattern = r"```(?:html|HTML)\s*\n(.*?)```"
    match = re.search(pattern, text, re.DOTALL)
    if match:
        return match.group(1).strip()

    # Fallback: try any code block
    pattern_fallback = r"```\s*\n(.*?)```"
    match = re.search(pattern_fallback, text, re.DOTALL)
    if match:
        return match.group(1).strip()

    raise ValueError("Could not extract HTML code block from Gemini response")

The fallback pattern catches cases where the model ignores the html language specifier but still wraps code in backticks. If both fail, we raise an error rather than writing garbage to a file.

Now the core conversion function:

def screenshot_to_html(image_path: str, output_path: str = "output.html") -> str:
    """Convert a screenshot to an HTML file using Gemini 1.5 Flash."""
    genai.configure(api_key=os.environ["GEMINI_API_KEY"])

    b64_string, mime_type = encode_image_to_base64(image_path)

    model = genai.GenerativeModel("gemini-1.5-flash")

    response = model.generate_content([
        SYSTEM_PROMPT,
        {
            "inline_data": {
                "mime_type": mime_type,
                "data": b64_string
            }
        }
    ])

    html_content = extract_html_from_response(response.text)

    with open(output_path, "w", encoding="utf-8") as f:
        f.write(html_content)

    print(f"✓ HTML written to {output_path} ({len(html_content)} chars)")
    return output_path

Step 5: Validating Output with Playwright Screenshots

Generating code is one thing. Knowing it renders correctly is another. We'll use Playwright to open the generated HTML in a headless Chromium instance and capture a screenshot for visual comparison.

def render_and_screenshot(html_path: str, screenshot_path: str = "rendered.png"):
    """Render an HTML file in headless Chromium and capture a screenshot."""
    abs_path = Path(html_path).resolve()

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(viewport={"width": 1440, "height": 900})
        page.goto(f"file://{abs_path}")
        # Wait for any JS-driven layout to settle
        page.wait_for_timeout(1000)
        page.screenshot(path=screenshot_path, full_page=True)
        browser.close()

    print(f"✓ Validation screenshot saved to {screenshot_path}")

The full_page=True flag captures the entire scrollable page, not just the viewport. This matters for longer designs that exceed 900px.

Running the Full Conversion Script

Tie everything together with a CLI entry point:

def main():
    parser = argparse.ArgumentParser(
        description="Convert UI screenshots to HTML using Gemini 1.5 Flash"
    )
    parser.add_argument("screenshot", help="Path to the screenshot file")
    parser.add_argument(
        "-o", "--output", default="output.html",
        help="Output HTML file path (default: output.html)"
    )
    parser.add_argument(
        "--no-validate", action="store_true",
        help="Skip Playwright validation screenshot"
    )
    args = parser.parse_args()

    if "GEMINI_API_KEY" not in os.environ:
        print("Error: Set GEMINI_API_KEY environment variable")
        sys.exit(1)

    html_path = screenshot_to_html(args.screenshot, args.output)

    if not args.no_validate:
        render_and_screenshot(html_path, "rendered.png")

if __name__ == "__main__":
    main()

Usage:

# Basic conversion
python convert.py my-ui-mockup.png

# Custom output path, skip validation
python convert.py landing-page.jpg -o landing-clone.html --no-validate

Sensible Extensions

This script is a foundation. Here's where to take it next:

1. Batch processing: Wrap the conversion in a loop over a directory of screenshots. Useful for design systems where you have 50+ component screenshots.

2. Diff-based iteration: After generating output.html, take a Playwright screenshot, compute a pixel diff against the original, and feed the diff back to Gemini with a "fix the differences" prompt. This creates a feedback loop that converges toward pixel-perfect output.

3. Component extraction: Modify the prompt to output individual components ("just the navbar", "just the card") rather than full pages. Store results in a component library directory.

4. Tailwind conversion: Add a second LLM call that takes the raw HTML/CSS and converts it to Tailwind utility classes. This makes the output more practical for React/Vue projects.

5. Multi-model comparison: Run the same screenshot through Gemini Flash, Gemini Pro (if you have credits), and Claude Haiku (free tier via Anthropic Console). Compare outputs automatically using Playwright screenshots and structural similarity metrics. This is a lightweight version of the approach discussed in our piece on Building an OpenRouter That Learns: How Usage Data Can Optimize Model Selection.

Common Pitfalls and Fixes

Pitfall 1: Gemini returns text outside the code block.

Symptom: extract_html_from_response raises ValueError.

Fix: Tighten the prompt. Add "Do NOT include any explanation, summary, or commentary outside the code block" at the end. If it persists, increase the model's temperature to 0 (more deterministic output) by passing generation_config={"temperature": 0} to generate_content().

Pitfall 2: Generated HTML looks nothing like the screenshot.

Symptom: Validation screenshot is wildly different.

Fix: The model may be struggling with complex layouts. Try cropping the screenshot to a single component or section. Gemini Flash has a 1M token context window but still performs better on focused inputs. Also, ensure your screenshot resolution is at least 800px wide — tiny images lose detail.

Pitfall 3: Rate limiting (429 errors).

Symptom: google.api_core.exceptions.ResourceExhausted.

Fix: The free tier allows 15 requests per minute. Add a time.sleep(4) between batch requests. If you hit the daily token limit, wait 24 hours or upgrade to pay-as-you-go ($0.075 per 1M input tokens — still absurdly cheap).

Pitfall 4: Playwright can't find Chromium.

Symptom: playwright._impl._api_types.Error: Executable doesn't exist.

Fix: Run playwright install chromium again. On Linux servers, you may need additional system dependencies: playwright install-deps chromium.

FAQ

Q: Does this work with dark mode screenshots?

Yes. Gemini handles dark backgrounds and light text without issues. The generated CSS will include appropriate background-color and color properties. If the output looks washed out, check that your OS isn't applying a color profile to the screenshot.

Q: Can I use this for mobile app screenshots?

Absolutely. Set the Playwright viewport to a mobile resolution (e.g., 390x844 for iPhone 14) in the validation step. The generated HTML will be responsive by default if you add max-width constraints to your prompt.

Q: How does this compare to specialized screenshot-to-code tools?

Dedicated tools like screenshot-to-code (open-source, GPT-4 Vision) produce more polished output because they use a multi-step pipeline: they first segment the image into components, then generate code for each. Our single-pass approach is simpler and free-tier-compatible, but the output quality ceiling is lower. For a deeper look at how FDEs approach build-vs-buy decisions like this, see From Messy Customer Problem to Shipped Prototype in a Week: An FDE Playbook.

Q: What's the largest screenshot this can handle?

Gemini 1.5 Flash accepts images up to 20MB. Practically, screenshots above 4000px in any dimension may cause the model to miss fine details. For full-page screenshots of long scrolling pages, split them into overlapping segments and stitch the generated HTML.

Q: Can I use this in a CI/CD pipeline?

Yes. The script is headless and deterministic (with temperature=0). Add it as a step that validates design-to-code fidelity: take a Figma export screenshot, generate HTML, screenshot the result, and compute a similarity score. Fail the build if similarity drops below a threshold.

---

This pipeline is a concrete example of the tactical execution patterns FDEs use daily: identify a free-tier model, constrain its output with strict prompting, and validate programmatically. If you're interested in building more workflows like this — from WhatsApp support agents to Discord FAQ bots — the same decomposition-first mindset applies across every domain.