Document indexing settings

[seppo]

I couldn’t find info on what the initial settings for document indexing do. I work as an SEO consultant and run a lot of crawlers, to me the settings in the screenshot would mean that Cursor would only crawl URLs inside the /docs/ folder, but that doesn’t seem to be the case and it crawls the entire site - including case studies.

[seppo]

Since I can upload only one image per post, here’s a screenshot showing Cursor has indexed pages outside of the /docs/ folder.

[toe]

had a very similar issue. this might help

Adding Documentations that don't have index-page or sitemaps

TL;DR use xml-sitemaps.com to create a make-do site-map of the documentation’s homepage/entrypoint. IMPORTANT: At the end of the page you’ll find “view HTML sitemap”. use that as the Entrypoint of the documentation. (Cursor doesn’t recognize XML, which is made available officially by every website, which would’ve made this a lot easier) the prefix is the “repeating” element of the official URL, NOT xml-sitemaps.com Full version: some documentations do not come with an …/index.html which make…