Is there a way to debug what info is fed into the docs crawler? Im trying to add our internal docs and i dont think its working. Maybe because the site is a spa. Would love to know whats happening internally so I can just fix the docs

We plan on adding a visualization of what it is the model actually sees when using docs in a future release. We’ll also have better docs management and auditing of text is shown for each page For now, we use an HTML to markdown parser, then do n-gram deduplication across the crawled pages (to take …

What do you mean by client-side rendered?

Like its a create react app. So its all rendered using react. The html just has the script tag. So im guessing the cursor server wont execute the JavaScript.

@amanrs Thank you Aman, very helpful!

Are the requests made from the client running cursor or from a cloud-based service? Would it be possible to access non-public resources like Azure DevOps wikis or internally hosted sites?

I came here to mention that I think there might need to be some kind of user assisted check or something… I want to play with a VSCode extension for my workflow, so I entered the url of Extension API | Visual Studio Code Extension API … 10 mins later it’s still learning. It’s following every link …

[image] Fardeem: Ahh so if the docs are client side rendered its not gonna work? They are rendered client side! We use puppeteer to control Chromium and run the Javascript.

Is there currently any way to control what links get followed? I’m trying to index Modules - io but it appears Cursor doesn’t follow the links. If I want it to index the actual docs for each module I have to do each one individually, and there are a lot of modules. BTW Cursor is absolutely amazing …

I still can’t get cursor to index Modules - effect It won’t follow the links for whatever reason. This makes it more or less impossible to use cursor features with my codebase. Without the docs it gives completely wrong and non-sensical answers about effect code. It uses APIs that don’t exist, impo…

I still can’t get it to index the Effect docs which really limits the usefulness of Cursor. Without the docs I get wildly incorrect answers from the AI about any effect code (which is most of my codebase).

Hey everyone! Sorry to resurrect a “dead” thread, but would love to bring the discussions back around here… I took am quite seriously struggling to index some documentation - it’s not clear to me what the issues are, however, and debugging/fixing the issues around docs are tough/opaque, at best… A…

How does docs crawling work

Discussions

truell20 (Michael Truell) September 5, 2023, 9:42pm 10

They are rendered client side! We use puppeteer to control Chromium and run the Javascript.

Which format is better for custom docs?

Topic		Replies	Views
More details on how the @docs work Help	4	1426	September 19, 2024
Indexed 0 pages Bug Reports	15	464	December 21, 2025
How to index Three.js doc? Bug Reports	4	634	September 15, 2024
Why is adding custom docs so hit or miss with Cursor? Bug Reports	5	1180	October 3, 2024
Cursor @Docs feature STILL BROKEN Bug Reports	31	667	February 21, 2026

How does docs crawling work

Related topics