Which format is better for custom docs?

[litecode]

Hi @ronenen ,

Your post made me think of these posts, hopefully they are relevant?

How does docs crawling work

We plan on adding a visualization of what it is the model actually sees when using docs in a future release. We’ll also have better docs management and auditing of text is shown for each page For now, we use an HTML to markdown parser, then do n-gram deduplication across the crawled pages (to take out useless info like navbars). Finally, we use some simple chunking heuristics to break down a page into multiple ~500 token chunks. The library name is node-html-markdown if you’d like to see what …

How does docs crawling work

They are rendered client side! We use puppeteer to control Chromium and run the Javascript.

Were you specifically wanting to know more about the suggested approach to linking to GitHub repo’s? If so, I think I have bookmarked a few posts about that topic that I could look for if relevant.