There is an older thread here that is still the clearest write up of the neighbouring problem I have found, “Getting screenshots into Cursor is driving me a bit crazy” (topic 162024). Reading it back, something struck me. Every answer in it, including the good ones about marking the element and sending the intent with the image, assumes I am the one who goes and looks. I take the shot, I crop it, I say what I meant.
The point there that stuck with me most was about the capture half being solved by shortcuts, while what still costs time is the agent guessing which element you meant and picking a neighbour. That is the same failure I keep hitting, just at a different scale. When my agent needs to know what an existing flow looks like, it does not ask me for a screenshot. It reads my code and infers. Two days ago I found ten feature flags in my own app whose registry said dark in production while a runtime override had had them live for weeks, so for over a month the source would have described screens nobody was seeing.
What broke the screenshot workflow for me was volume. The agent adds more screens in a week than I am willing to go and capture by hand.
So I went at the other end and let the agent go look at the deployed app itself, signed in, keeping what it saw with dates, so “show me the referral flow” is answerable without me opening anything.
Two questions for this forum:
When your agent needs to know what a screen looks like right now, does it look, or does it read the code and infer? Has that inference ever sent it the wrong way?
Has anyone tried taking themselves out of the capture loop entirely, or is a marked up shot you took on purpose always going to beat something the agent went and fetched?
Hey, great framing of the request. This is a real pain point when an agent treats the code as the source of truth for the UI, but the runtime diverges. Your feature flags example illustrates it well.
On your first question, the agent has a built-in Browser tool. It opens as a panel right inside Cursor, and the agent can navigate the running or deployed app itself, click through it, and capture what it actually sees instead of guessing from the source. Docs: Browser | Cursor Docs and the overview: Overview | Cursor Docs. It is basically the “the agent goes and checks for itself” pattern you described, but built in and supported out of the box, so you do not need to build custom glue.
On “inference going off the rails”, yes, it is the same class of problem as in 162024 with picking a neighboring element. When the context is not anchored to what is really on screen right now, the model fills gaps with guesses. The Browser tool covers the “show me the referral flow” case without you having to open the app yourself.
On the second question, it is worth hearing what the community thinks. My practical take is that an agent-fetched screenshot is great for “what is true right now”, while a marked-up screenshot you capture manually still wins when it matters to communicate intent, like “this element goes here”. These are two different tools for different jobs, not an either or. Curious what you landed on after removing yourself from the capture loop. Were there cases where the lack of your markup hurt accuracy?
It feels like there’s a broader pattern here: agents often treat the repository as the source of truth, even when production has drifted. Runtime config, feature flags, environment variables and manual changes can all make the live system very different from what the code suggests.
I’m curious, once you had the agent inspect the deployed app directly, did you trust it to keep doing that automatically, or did you still feel the need to verify what it was using as its source of truth?
Agent-driven inspection breaks down on anything transient: a hover state, a toast that disappears after two seconds, a validation message that only shows up after a specific keystroke sequence. Getting the agent to script its way into that exact frame usually takes longer than just taking the screenshot by hand at the right moment. I still keep manual capture for timing-sensitive states and let the agent handle the steady pages on its own.
Thanks, both of you pointed out exactly where this approach breaks.
@ActionGate I wouldn’t give the agent full “trust and don’t verify.” The Browser tool is great for answering “what’s on the screen right now” because it looks at the real runtime, not the code. But its source of truth is still a specific session: what was logged in, which flags or overrides were active at that moment. So I treat an agent screenshot as a fresh snapshot of the current state, not as a permanent truth. For critical flows, it’s still worth double-checking which environment the agent was looking at.
@bickov I completely agree, and that’s a good distinction. Transient states like hover, a toast that lasts two seconds, or validation after a specific sequence are exactly the cases where scripting the agent to catch the right frame costs more than just taking the screenshot manually at the right moment. A sensible split is to give steady pages to the agent through the Browser tool, and capture timing-sensitive states manually with clear intent labels.