AI-Driven GUI / End-to-End Testing Agent for Desktop Applications

Ideas Feature Requests
1

Feature request for product/service

Cursor IDE

Describe the request

Hi everyone,
I’m currently developing desktop software with Rust (using frameworks like Tauri, Iced, or egui, among others), and I came up with an idea for a feature that would massively ease everyday development work and fits perfectly with Cursor’s vision.

The Problem:
While unit tests work great for pure logic (especially in Rust), testing the user interface is often a real pain point. Automated UI tests for desktop apps usually require writing fragile scripts that break with the smallest design change. In the end, it often comes down to manually clicking through the application after every major build to check whether the menus work, the UX is right, and no unexpected errors or crashes occur.

The Idea: A Visual UI Testing Agent in Cursor:
Since Cursor already relies heavily on AI-driven development, an integrated “end-to-end” (E2E) testing agent would be the next logical step. It would be fantastic if Cursor could independently run the compiled software on command and test it through clicks.
What the workflow could look like: You give the AI a simple prompt, for example:
“Compile the app, open the settings menu, change the language to English, and check whether the UI freezes or the program throws errors in the terminal.”
Cursor could then launch the application and use computer vision or the operating system’s accessibility APIs to analyze the window contents, recognize buttons, simulate clicks, and cross-check the visual feedback against the log output.

The Benefits at a Glance:

  • Massive time savings: The AI takes over the tedious and time-consuming work of a manual QA tester.
  • Less maintenance: Developers no longer need to write and maintain error-prone UI test scripts.
  • Full focus on logic: You can concentrate on coding while Cursor systematically runs through edge cases in the background or checks whether all buttons across the various menu levels actually respond.

For the entire desktop ecosystem — completely independent of the language used — a feature like this would be an absolute game changer and a unique selling point for cursor.

5

Hey, thanks for the detailed feature request. The request is clear: an agent that launches the built desktop app itself and clicks through the UI while checking visual feedback against logs.

This is already possible in part but it depends on the app type.

  • For web-UI the Browser tool covers it. The agent can navigate, click, type, scroll, take screenshots and read console plus network traffic. That matches the open, click around and check for errors scenario. Details here: Browser | Cursor Docs
  • For native desktop apps like Tauri, Iced or egui there is no built-in computer-use in the local IDE agent yet. The current workaround is to connect a computer-use MCP server for screenshots and control via accessibility API or click simulation. Then the agent can drive the GUI from the prompt. How to set up MCP: Model Context Protocol (MCP) | Cursor Docs

I logged the request for native desktop GUI testing. This direction is on the radar but I can’t give an ETA. If an update appears I’ll reply in the thread.