Need more information about how doc indexing works

What is going on when documents are indexed?

  • I must have a local vector-db for the docs.
  • My hope is that the docs are being turned into AST before being chunked
  • When I add a doc to my prompt, how are you taking the prompt and doing a query of the db?

I would just like some more information on this process. I have been unable to find it on the cursor site or forum. I feel like cursor does not do a good job of finding information in the docs and I am trying to figure out why. I have found a few people that are storing their documents under a .notes folder and referencing the document in their prompts instead of using Docs, but they will loose the benefits of the vector-db and it will only work for small docs (I guess, because I do not even know what happens when I link to a file in my prompt)

It would be nice if what was going on was more exposed and I had the ability to modify the behavior.

FYI: I am sure you already know about it, but this Docling project seems interesting for preparing docs for adding to the vector-db: GitHub - DS4SD/docling: Get your documents ready for gen AI

Hey, while I can’t provide details on how document indexing works, I can tell you that the indexing is done on our servers, and the latest version of documentation is shared for all users (given the same input URLs) so it should always be up to date!

Regarding its ability to pull relevant context from the docs, we are always working to try to improve this, but it is often better to provide the AI with the exact page of documentation you want it to read, to ensure it has the correct context.

The @docs is more useful when there could be multiple, relevant sections in the documentation that the AI might need.

1 Like