Which format is better for custom docs?

[ronenen]

Hi good people,
Does Cursor have some “filter” to remove toc navigation bar from htmls for the indexing (assuming it uses semantic vectors?) Should I rather link to the repsitory of reStructuredText files that the htmls are generated from?
Thanks

[litecode]

Hi @ronenen ,

Your post made me think of these posts, hopefully they are relevant?

How does docs crawling work

We plan on adding a visualization of what it is the model actually sees when using docs in a future release. We’ll also have better docs management and auditing of text is shown for each page For now, we use an HTML to markdown parser, then do n-gram deduplication across the crawled pages (to take out useless info like navbars). Finally, we use some simple chunking heuristics to break down a page into multiple ~500 token chunks. The library name is node-html-markdown if you’d like to see what …

How does docs crawling work

They are rendered client side! We use puppeteer to control Chromium and run the Javascript.

Were you specifically wanting to know more about the suggested approach to linking to GitHub repo’s? If so, I think I have bookmarked a few posts about that topic that I could look for if relevant.

[ronenen]

Thank you @litecode for your answer. I guess this makes html documentation the way, since Cursor takes out the navigation bars.