# Application Architecture **URL:** \ **Category:** Discussions\ **Created:** [September 3, 2023, 2:19pm UTC](https://forum.cursor.com/t/application-architecture/328 "2023-09-03T14:19:09Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![ash](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@ash](https://forum.cursor.com/u/ash)\ **Post date:** [September 4, 2023, 8:34am UTC](https://forum.cursor.com/t/application-architecture/328/2 "2023-09-04T08:34:51Z")
> [@How does docs crawling work](https://forum.cursor.com/t/how-does-docs-crawling-work/264/3): > > We plan on adding a visualization of what it is the model actually sees when using docs in a future release. We’ll also have better docs management and auditing of text is shown for each page For now, we use an HTML to markdown parser, then do n-gram deduplication across the crawled pages (to take out useless info like navbars). Finally, we use some simple chunking heuristics to break down a page into multiple ~500 token chunks. The library name is node-html-markdown if you’d like to see what … Doc Crawling Explained --- _[View the full topic](https://forum.cursor.com/t/application-architecture/328)._