Another custom documentation issue

Hi everyone. I’ve been trying to get Cursor to reference custom docs that are publicly hosted on the web, but I get the “0 pages indexed” issue no matter what I try. I even followed the suggestion of generating a sitemap via xml-sitemaps.com and referencing that to no avail.

Next, I thought maybe I could create a cursor rule that includes a site directory for the docs but when referencing the rule and asking about the docs, Cursor seems to be unable to access the URLs directly and instead hallucinates the answer. Finally, I tried downloading all the docs to markdown format and just throwing that into a rule but that just gave a context limit error because of the size of the markdown.

I love Cursor, but this issue is pretty frustrating and I hope they can fix it. In the meantime, can anyone suggest any workarounds?

Hey, can you share the link to the documentation?

cc @danperks

Screenshot 2025-03-03 at 21.34.17

Unfortunately, it appears the docks have some kind of anti-bot protection behind them, so they are not scrapable yet by Cursor.

We are hoping to improve this soon so we can add support for these kinds of websites!