Adding Documentations that don't have index-page or sitemaps

Showcase
1

TL;DR

  1. use xml-sitemaps.com to create a make-do site-map of the documentation’s homepage/entrypoint.
  2. IMPORTANT: At the end of the page you’ll find “view HTML sitemap”. use that as the Entrypoint of the documentation. (Cursor doesn’t recognize XML, which is made available officially by every website, which would’ve made this a lot easier)
  3. the prefix is the “repeating” element of the official URL, NOT xml-sitemaps.com

Full version:
some documentations do not come with an …/index.html which makes a great Entrypoint for Cursor.
To make matters worse, some websites like docs.langflow.org do not use traditional hierarchy in their URL. (by hierarchy i mean, the links are not in example.com/page/sub-page/.. format)
For example, in the case of langflow, the URL structure is example.com/Topic-SubTopic-SubSubTopic.. which seems to be an unfamiliar format for Cursor’s bot (atleast at the time of writing this)
Due to which my attempts at indexing Langflow’s documentation had Cursor forcing itself to look for links with a hierarchical format which I mentioned earlier. So it ended up hallucinating links which don’t exist, and indexed 404 pages which were in hierarchical format. Yet, it hallucinated answers with confidence. Classic AI.
A sitemap can fix this issue, but when there isn’t one, you gotta make your own. Most websites keep a xml sitemap for the search-engines (example.com/sitemap.xml)
But Cursor doesn’t work with XML links. That’s where XML-sitemaps.com comes in. It not only crawls the given link to make a sitemap, it also gives a HTML version of the result which can be used as an Entrypoint with Cursor.

Edit: the hierarchical URL structure is called path-based or directory-style URL. the one with the hyphen can be called flat/delimited URL, although gpt4 said it’s not a recognized term

5 Likes
2

Can you explain this again? Right now I have used xml-sitemaps.com to generate the HTML link. I pasted that into the add new doc dialog. It shows indexed but displays 0 pages?

I set prefix as official url

Are there instructions on how can generate docs that are fully compatible with how they are indexing the data?

3

for example, in case of example.com/xyz/abc
use only example.com as prefix
the HTML-XML-sitemap of example.com as the Entrypoint

is that what you did?

4

Bro. You. Rule.