Screenshot-based UI annotations that feed into chat context (Antigravity-style)

Ideas Feature Requests
1

Feature request for product/service

Cursor IDE

Describe the request

Hi Cursor team :waving_hand:

I’d love to see a feature similar to Google Antigravity’s screenshot “artifacts” / comments system, but inside Cursor.

What I’m asking for

A way to:
1. Capture a screenshot of the running app (or paste/upload one).
2. Draw a rectangle / highlight a specific region on the screenshot.
3. Attach a short comment to that region.
4. Have both the screenshot and the selected region + comment automatically sent into the current chat context.

So instead of writing “the button in the top-right looks off” and hoping the model understands, I could literally point to the exact place in the UI and say:

“Increase spacing between these two elements by 8px and align them vertically.”

The model would see the image, the selected region, and the text, and then propose concrete code changes.

Why this would be super useful

•	Frontend + UI work becomes much more precise. Small visual issues (1–2 px misalignment, wrong font weight, inconsistent spacing, hover states, etc.) are much easier to show than describe.

•	Better collaboration between devs, designers and PMs. Designers/PMs could highlight issues visually and let Cursor suggest the corresponding code changes, instead of writing long text descriptions.

•	Faster feedback loops When working on a feature, I could quickly screenshot the current state, mark 3–4 areas with comments (like in Figma/Antigravity), and ask Cursor: “Apply all of these tweaks in the code.”

Nice-to-have details

•	Multiple annotations per screenshot, each treated as a separate “comment” in the chat thread.
•	Being able to say in chat: “Fix all issues from this screenshot” and have the model iterate through all annotations.

In short: screenshots + region-based comments that become first-class context for the AI, focused on improving front-end and UI workflows. I think this would massively improve the experience for anyone doing visual/frontend work in Cursor.

2

This is the actual gap. Annotating is half of it. The half that matters is what gets sent: not the pixels, but which element you marked, where it sits, and the text on it, so the agent goes straight to it instead of reading the whole frame. I run something like this outside Cursor right now and it’s the difference between one prompt and five. Native would be better.

3

@bickov nailed the main point. The value isn’t in the image itself, it’s in what you highlighted: what the element is, where it is, and what text is on it.

That part is already partly covered by Design Mode in the browser Agents Window. When you select an element in the preview, the chat sends a reference to the element itself, not just a screenshot, so the agent can go straight to it. Guide: Design Mode | Cursor Docs

What we don’t have yet is a full annotation layer on top of an image: multiple rectangles or comments on one screenshot, plus a command like “go through all notes.” That’s a valid request, I’ll pass it to the team. I can’t share an ETA yet.

Since you’re running a similar workflow outside Cursor, can you describe what exactly gets sent to the agent from the annotation. Is it only coordinates plus text, or does it also include the selector or DOM info for the element. Details like that would really help us shape a native version.

4

Great request, this is basically the workflow I live in. Worth knowing you can get most of it today instead of waiting. Cursor’s Design Mode already does region-select for web. For the rest, native apps, desktop, or your “mark 3-4 areas then fix all” case, what works is a capture step that saves each marked region plus its comment as structured text the agent reads, so it acts on the exact elements, not a flat image. I build one for that (SlimSnap), but the general pattern, region + intent as data, is what I’d push Cursor to adopt natively. The multi-annotation “fix all from this screenshot” part especially, that’s the killer.

6

Hey @bickov, that’s a helpful framing: region + intent as data, not a flat image. That’s exactly the idea we’re exploring for the native version.

One thing from my question above is still open, and it’s key for implementation. When you save a marked region plus a comment as structured text, what exactly does the agent get for the element itself? Is it only the bounding box coordinates plus the text, or do you also pass a CSS selector, DOM path, or some stable element identifier?

The difference matters. On web we can anchor to an element via Design Mode, but on desktop or native apps there’s no stable selector. I’m curious how you handle anchoring there. If you can share how this part works on your side, it’d really help us design the fix all from this screenshot flow.

Status update: the request for a multi-annotation layer has been passed to the team, no ETA yet.

7

Good question, and it’s the crux of the native case. On native there’s no stable selector, so I don’t fake one, the anchor is descriptive, not referential. For each marked element the agent gets a normalized bounding box (survives resize), the OCR’d text of that element, its type and color, the nearby text as disambiguating context, and the intent/callout. So instead of “selector #x,” it’s “the element that reads Save, here, this color, change it to do Y.”

That’s usually enough because the OCR text tends to match a string literal or label in the source, so the agent can find it in code without a selector. On web you can layer a real selector on top via Design mode, but the descriptive anchor works on both, which is the point for native.

Full shape is the open schema (MIT): github.com/bickov/slimsnap-schema. If your native version emits or reads something close to it, the tooling gets shared instead of everyone inventing their own. Happy to go deeper on the multi-annotation part, the “fix all” case is just an array of these element+intent objects.

9

Thanks, that was exactly the missing piece. Using a descriptive anchor instead of a selector reference makes sense for native apps, where there isn’t a stable selector. The part about the element’s OCR text usually matching a string literal or label in the source is especially useful, so the agent can find the element in the code without a selector. On the web, you can add a real selector on top of that via Design Mode, and the descriptive layer works in both cases, which is a solid common baseline.

I also noted the “fix all” idea as an array of objects with element plus intent, that’s a clear model for multi-annotation.

If there’s an update, I’ll post it here.