Cursor cannot index docs sites

Bug Reports
1

1

Consider docs at https://arktype.io/docs/intro

There is no top-level HTML page at the /docs path as the manual is split into sections:

  • https://arktype.io/docs/intro
  • https://arktype.io/docs/section1
  • https://arktype.io/docs/section2
  • etc

Now two things make using this docs site impossible in Cursor:

  • There is currently no way to list URLs patterns in the Cursor Settings.
  • Cursor’s crawler doesn’t follow “horizontal” links for some reason.

2

Thinking that maybe Cursor doesn’t visit sibling URLs (i.e. those at https://arktype.io/docs/*) when provided with a more specific one (i.e. https://arktype.io/docs/intro) I tried to add the root site page instead:

image
image807×370 15 KB

Cursor indexed 6 pages:

image
image507×277 15.2 KB

which were:

  • /
  • /docs/intro/* (4 pages in total)
  • /discord

but it didn’t visit the entire /docs/* area.

As you see, now that it’s not constrained with a specific URL it still doesn’t follow horizontal links.

UPDATE. Let me check if the site provides correct HTML…
UPDATE. Yes, it does. I disabled JavaScript and received perfect page with all the links in place. So it’s definitely a problem of docs-indexing crawler of Cursor.