# Adding Documentations that don't have index-page or sitemaps **URL:** \ **Category:** Guides\ **Created:** [September 5, 2024, 9:02pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171 "2024-09-05T21:02:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![toe](https://avatars.discourse-cdn.com/v4/letter/t/8797f3/32.png) [@toe](https://forum.cursor.com/u/toe)\ **Post date:** [September 5, 2024, 9:02pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/1 "2024-09-05T21:02:33Z")
TL;DR 1. use [xml-sitemaps.com](http://xml-sitemaps.com) to create a make-do site-map of the documentation’s homepage/entrypoint. 2. IMPORTANT: At the end of the page you’ll find “view HTML sitemap”. use that as the Entrypoint of the documentation. _(Cursor doesn’t recognize XML, which is made available officially by every website, which would’ve made this a lot easier)_ 3. the prefix is the “repeating” element of the official URL, NOT `xml-sitemaps.com` Full version: some documentations do not come with an **../index.html** which makes a great Entrypoint for Cursor. To make matters worse, some websites like [docs.langflow.org](http://docs.langflow.org) do not use traditional hierarchy in their URL. (by hierarchy i mean, the links are not in `example.com/page/sub-page/..` format) For example, in the case of langflow, the URL structure is `example.com/Topic-SubTopic-SubSubTopic..` which seems to be an unfamiliar format for Cursor’s bot _(atleast at the time of writing this)_ Due to which my attempts at indexing Langflow’s documentation had Cursor forcing itself to look for links with a hierarchical format which I mentioned earlier. So it **ended up hallucinating links which don’t exist** , and indexed 404 pages which were in hierarchical format. Yet, it hallucinated answers with confidence. Classic AI. A sitemap can fix this issue, but when there isn’t one, you gotta make your own. Most websites keep a xml sitemap for the search-engines `(example.com/sitemap.xml)` But Cursor doesn’t work with XML links. That’s where `XML-sitemaps.com` comes in. It not only crawls the given link to make a sitemap, it also gives a HTML version of the result which can be used as an Entrypoint with Cursor. Edit: the hierarchical URL structure is called path-based or directory-style URL. the one with the hyphen can be called flat/delimited URL, although gpt4 said it’s not a recognized term ---
**Author:** ![adgower](https://sea3.discourse-cdn.com/cursor1/user_avatar/forum.cursor.com/adgower/32/4537_2.png) [@adgower](https://forum.cursor.com/u/adgower)\ **Post date:** [September 12, 2024, 2:15pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/2 "2024-09-12T14:15:43Z")
Can you explain this again? Right now I have used [xml-sitemaps.com](http://xml-sitemaps.com) to generate the HTML link. I pasted that into the add new doc dialog. It shows indexed but displays 0 pages? I set prefix as official url Are there instructions on how can generate docs that are fully compatible with how they are indexing the data? ---
**Author:** ![toe](https://avatars.discourse-cdn.com/v4/letter/t/8797f3/32.png) [@toe](https://forum.cursor.com/u/toe)\ **Post date:** [September 13, 2024, 7:43am UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/3 "2024-09-13T07:43:09Z")
for example, in case of `example.com/xyz/abc` use only `example.com` as **prefix** the `HTML-XML-sitemap of example.com` as the **Entrypoint** is that what you did? ---
**Author:** ![ron](https://sea3.discourse-cdn.com/cursor1/user_avatar/forum.cursor.com/ron/32/5201_2.png) [@ron](https://forum.cursor.com/u/ron)\ **Post date:** [September 18, 2024, 2:07am UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/4 "2024-09-18T02:07:58Z")
Bro. You. Rule. ---
**Author:** ![Hrnkas](https://sea3.discourse-cdn.com/cursor1/user_avatar/forum.cursor.com/hrnkas/32/24195_2.png) [@Hrnkas](https://forum.cursor.com/u/Hrnkas)\ **Post date:** [February 21, 2025, 9:23pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/5 "2025-02-21T21:23:39Z")
Can anyone tell me what is wrong with my documentation pages: [Future-C Documentation](https://futurefactory-software.com/scriptdocs/index.html) sitemap: [futurefactory-software.com Site Map - Generated by www.xml-sitemaps.com](https://futurefactory-software.com/scriptdocs/sitemap.html) cursor refuses to index them (indexed 0 pages) ---
**Author:** ![OhoyCaptain1](https://avatars.discourse-cdn.com/v4/letter/o/e0b2c6/32.png) [@OhoyCaptain1](https://forum.cursor.com/u/OhoyCaptain1)\ **Post date:** [March 1, 2025, 1:12pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/6 "2025-03-01T13:12:48Z")
Bump on this topic. Is this a broken solution (or a last hail Mary attempt)? I tried if for example with yahoo finance api and also got not luck despite being able to generate the site-xml without issues. ---
**Author:** ![benjaminfortunato](https://sea3.discourse-cdn.com/cursor1/user_avatar/forum.cursor.com/benjaminfortunato/32/17016_2.png) [@benjaminfortunato](https://forum.cursor.com/u/benjaminfortunato)\ **Post date:** [April 7, 2025, 12:54pm UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/7 "2025-04-07T12:54:29Z")
I’m running into an issue with indexing being blocked: **Starting URL: [https://github.com/serverless/serverless/tree/main/docs](https://github.com/serverless/serverless/tree/main/docs)** **Updated on:** April 7, 2025, 12:53, 0 pages #### Your Sitemap is empty docs - Blocked by robots.txt file Looks like your website pages are blocked from indexing, please try our [Search Engine Robot Simulator](https://www.xml-sitemaps.com/se-bot-simulator.html) tool to diagnoze possible issues. ---
**Author:** ![system](https://sea3.discourse-cdn.com/cursor1/user_avatar/forum.cursor.com/system/32/99069_2.png) [@system](https://forum.cursor.com/u/system)\ **Post date:** [June 18, 2026, 10:37am UTC](https://forum.cursor.com/t/adding-documentations-that-dont-have-index-page-or-sitemaps/15171/8 "2026-06-18T10:37:16Z")
Closed on June 18, 2026, 10:37am UTC