I’m familiar with the “Cursor Settings → Docs” feature for indexing publicly available URLs that contain documentation. Unfortunately that only seems to allow indexing publicly available URLs. Our team has many gated documentation sources for private APIs that are not public, along with various Confluence standards docs about the different types of code we write.
How are other teams indexing this private information and making it available to Cursor?
We’ve considered trying to reverse engineer what IPs the Cursor indexing spider uses and making those docs available only to those IPs, but that’s generally not a good idea from a security perspective.
Should we be scraping these docs ourselves and curating them into Markdown format and making available in each repo that our engineers are working on? One giant .cursorrules file?
This seems like a common business use case and have been surprised to not come across any documentation or discussion of others doing this in my own searching.
Only public URLs are available currently. What you can do is just like you say, convert these to markdown then store them in a folder somwhere in you codebase.
Then it’s very easy to just include that in the context by writing @docs/your-private-doc.md in the prompt.
Ultimately it boils down to whether we treat each individual engineer using Cursor as independent to one another and require and hope that they setup their environment in the most efficient way… verus, setting the defaults in a sane way that works for our environment with the tools they need already at hand. There’s significant business value in streamlining that process and getting our docs that are locked up into LLM tools to make them as useful as possible.
Hey! I’m in the process of encouraging our company to start using Cursor, but same issue - we have lots of documentation on Confluence and it would be great for Cursor to be able to read it! That would make the sell much easier
This would be a huge unlock at my company too. We have a lot of developer documentation stored on a custom docs site built using Backstage, but can’t currently access it from Cursor.
We can’t make the documentation public, and there’s too much of it for it to be feasible to add the docs to our hundreds of repositories as keeping it up to date would be very challenging.
Having the ability for Cursor to vector search through our private docs site would be big deal for us!
I created a cloudflare worker which has access to some of our private docs. It then exposes those docs at URL.com - MediaOptions it’s working although i’m not sure if cursor shares those docs with the rest of the users (outside of our company) . do you know @ericzakariasson ?
Please dear cursor team I need this. I just set up a whole sphinx documentation pipeline to enable cursor to index one of our internal python packages and now I discovered cursor is only able to crawl public pages. What a downer! Cant you just initiate the crawling from the users machine?
. do you know