OpenAI Presence: The Desktop Operator That FDEs Can Actually Use Today

What Actually Happened: The Plain Announcement

OpenAI dropped a new research project called Presence. It's not a product you can buy. It's a direction. The core idea: an AI agent that can see your screen, move your mouse, type on your keyboard, and navigate applications—exactly like a human operator would. Think of it as giving an LLM direct motor control over your desktop environment.

The demo showed Presence handling multi-step workflows across different apps: pulling data from a web dashboard, pasting it into a spreadsheet, formatting it, then composing an email with the results. No APIs. No custom integrations. Just screen pixels, mouse coordinates, and keyboard events.

This matters because it sidesteps the integration tax that kills most automation projects. Every enterprise tool has an API these days, but they're inconsistent, rate-limited, poorly documented, or locked behind procurement. Presence says: "Forget the API. I'll just use the UI."

The technical underpinnings are a vision-language model that processes screenshots, a reasoning layer that plans actions, and an execution layer that translates those plans into OS-level input events. OpenAI calls it an "operator"—a term that signals this isn't a chatbot with tools, but an autonomous agent that drives the computer.

Why This Matters for Engineers and FDEs

Forward Deployed Engineers live in the gap between what software can do and what customers actually need. You're the person who gets dropped into a customer's environment and told: "Make our systems talk to theirs." Often, there's no API. There's a legacy mainframe with a green-screen terminal, or a SaaS tool that only exports CSVs on Tuesdays.

Presence-style agents change the calculus. Instead of spending three days writing a brittle screen-scraper for that legacy system, you could potentially point an operator at it and say: "Every morning at 8 AM, log in, download yesterday's transactions, and push them to this webhook."

Here's the FDE-specific value prop:

• Zero-integration deployments: You don't need the customer's IT team to open firewall ports or provision API keys. The agent operates at the UI layer, which is already accessible.
• Legacy system bridging: That AS/400 terminal emulator your manufacturing customer still runs? Presence doesn't care. It sees pixels and clicks buttons.
• Rapid prototyping in customer environments: Before you commit to building a proper integration, you can validate the workflow with an operator in hours, not weeks. This is straight out of the FDE customer prototype playbook.
• Reduced context-switching: An operator can watch a customer's workflow, replicate it, and then you can optimize from a working baseline rather than starting from documentation that may be wrong.

But here's the thing: Presence itself isn't publicly available. What OpenAI showed is a research preview. So the real question for working engineers is: can you build something similar today with what's already on your laptop?

Architecture: How a Desktop Agent Actually Works

Let's break down the components. A desktop operator has three layers:

1. Perception Layer (The Eyes) Takes periodic screenshots of the desktop. A vision-language model (GPT-4V, Claude 3.5 Sonnet, or local options like Llama 3.2 Vision) processes these to understand what's on screen: buttons, text fields, data tables, error messages. This is the hardest part—screen parsing is messy, resolutions vary, and UI elements move around.

2. Planning Layer (The Brain) Given a high-level goal ("extract Q3 sales data and email it to finance"), the planner breaks it into atomic actions. Each action is something like: click(button="Export CSV"), type(text="Q3", field="Date Range"), wait_for(element="Download complete"). The planner also handles errors—if a popup appears, it needs to recognize and dismiss it before continuing.

3. Execution Layer (The Hands) Translates planned actions into OS-level events. On macOS, that's Core Graphics and Accessibility APIs. On Windows, it's the Win32 API or UI Automation. On Linux, xdotool or similar. This layer also handles timing—waiting for UI transitions, handling loading spinners, and retrying on failure.

Here's what the flow looks like:

Notice the error handler. That's not optional. Desktop automation without error handling is a demo, not a tool. Your agent will encounter unexpected dialogs, network timeouts, OS notifications, and apps that decided to update themselves. The difference between a toy and something you'd run in production is how gracefully it recovers.

How to Try or Build Your Own Today

Presence isn't available, but you can build a functional desktop operator this weekend. Here's the stack I'd recommend for an engineer who wants to ship something real:

Option A: The Quick Win (Python + Claude/GPT-4V + PyAutoGUI)

# Minimal desktop agent loop
import pyautogui
import base64
from anthropic import Anthropic
from io import BytesIO

def capture_screen():
    screenshot = pyautogui.screenshot()
    buffered = BytesIO()
    screenshot.save(buffered, format="PNG")
    return base64.b64encode(buffered.getvalue()).decode()

def plan_action(screenshot_b64, goal):
    client = Anthropic()
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64}},
                {"type": "text", "text": f"Goal: {goal}. Return the next action as JSON: {{'action':'click'|'type'|'wait'|'done', 'x':int, 'y':int, 'text':str}}"}
            ]
        }]
    )
    return response.content[0].text

# This is a skeleton—real implementations need coordinate scaling, error recovery, and safety guards

This gets you a single-step agent in ~50 lines. It takes a screenshot, sends it to a vision model, gets back a mouse coordinate, and clicks. Loop that with a goal and you've got a basic operator.

Option B: The Production-Grade Approach

For something you'd actually run in a customer environment, you need more. Specifically:

• Coordinate normalization: Screenshots get resized for the model. You need to map model-output coordinates back to actual screen coordinates based on DPI scaling and image dimensions.
• Action validation: Before clicking, verify the target element exists. Use OCR on a small region around the target coordinates to confirm the expected text is there.
• Human-in-the-loop for destructive actions: If the agent is about to send an email, delete files, or submit a form, pause and ask for confirmation.
• Session recording: Log every screenshot and action for debugging. When the agent does something wrong at 3 AM, you need to know why.

If you've built a codebase Q&A tool with local models, you already have the patterns for this. Desktop agents are the same architecture—retrieval, reasoning, action—just with pixels instead of text chunks.

Option C: Browser-Only Agents

If full desktop control feels like too much surface area, start with browser automation. Playwright gives you structured access to the DOM, which is vastly easier than raw pixel parsing. You can build a job-application autofill agent that navigates forms, fills fields, and submits—all without touching OS-level input APIs. The same principles apply: perceive, plan, execute, recover.

A Balanced Take: The Sharp Edges

I'm excited about this direction, but let's be honest about where it breaks.

Security is the elephant in the room. You're giving an LLM keyboard and mouse control. Even with sandboxing, a hallucinated action can do real damage. "Click the export button" becomes "click the delete button" because the model confused two similarly-colored UI elements. Safety isn't a feature you bolt on—it has to be architectural. Every action needs validation, every destructive operation needs confirmation, and the agent needs hard boundaries (no rm -rf, no sending emails without review, no accessing files outside a whitelisted directory).

Reliability is still sketchy. Vision models misread text, especially at low resolutions or with unusual fonts. UI elements that look identical to humans (two "Submit" buttons on the same page) confuse models. Dynamic layouts break coordinate-based clicking. These aren't dealbreakers, but they mean you can't just set it and forget it. Expect to build monitoring and alerting around your agents.

Latency adds up. Each loop is: screenshot (200ms) → API call (1-3 seconds for a vision model) → action execution (100ms). A 10-step workflow takes 15-30 seconds. For batch automation, that's fine. For interactive use, it's painful. Local models help—a quantized Llama 3.2 Vision running on a MacBook can get loop times under 2 seconds.

The API-is-better argument. If the target application has an API, use it. APIs are faster, more reliable, and don't break when the vendor redesigns their UI. Desktop operators are for the long tail of tools that don't have APIs, or for environments where you can't get API access. Know when to use the right tool.

Cost at scale. Running a vision model every few seconds adds up. At current API prices, a single agent running 8 hours a day could cost $20-50/day in inference. That's cheap compared to a human, but expensive compared to a cron job hitting a REST endpoint. As with controlling reasoning effort in LLMs, you need to be intentional about when you pay for full vision reasoning versus cheaper, dumber checks.

FAQ: Safety, Access, and What Comes Next

Q: Is OpenAI Presence available to use right now? No. It's a research preview. OpenAI hasn't announced a release date, API, or pricing. What they showed is a direction, not a product.

Q: How is this different from RPA (Robotic Process Automation)? RPA tools like UiPath use brittle, rule-based selectors ("click the button with CSS class .btn-export"). Presence uses vision models that understand the meaning of the UI, not just its structure. That makes it more robust to UI changes but less predictable. RPA is deterministic; LLM-driven operators are probabilistic.

Q: Can I run this entirely locally? Yes, with compromises. A quantized Llama 3.2 Vision or a fine-tuned Florence-2 model can handle screen understanding on-device. Combine with xdotool (Linux) or AppleScript (macOS) for execution. The loop will be slower and less accurate than cloud models, but it works for constrained workflows. This is the same tradeoff you make when building a local meeting notetaker—local models give you privacy and zero latency, but lower quality.

Q: What about multi-monitor setups? Most current implementations only handle a single screen. Multi-monitor adds coordinate space complexity and dramatically increases the screenshot size (and thus API cost). It's solvable but not trivial.

Q: Will this replace Forward Deployed Engineers? No. It changes what you spend time on. Instead of writing screen scrapers and fighting with undocumented internal tools, you orchestrate agents and handle the edge cases they can't. The FDE role shifts from "I build the integration" to "I build the system that builds the integration." The weekly time audit already shows FDEs spending huge chunks on integration glue—operators eat that work. What remains is the high-judgment work: understanding customer needs, designing workflows, and handling the 5% of cases where the agent gets stuck.

Q: Where do I start if I want to build this skill set? Build a browser agent first. It's a constrained environment with excellent tooling (Playwright). Once you're comfortable with the perceive-plan-execute loop in a browser, graduate to desktop automation. The patterns transfer. Document everything—these are the projects that make your portfolio stand out, especially if you're targeting FDE roles where compensation is competitive and employers are looking for engineers who can ship in messy environments.

Bottom line: Desktop operators are real, they're buildable today, and they're going to change how FDEs work with customer environments. Presence is the polished vision. Your weekend project is the practical version. Ship it.