That link doesn’t really address anything regarding crawling behind a firewall.
Is it possible to use a one-shot-URL that gives access, get the documentation indexed, then block access? Would Cursor continue to attempt to crawl and eventually drop the index because it can’t access it?
We are already uploading all our code for indexing (albeit it gets “hashed” after indexing for lack of a better term), can we not do the same for our docs? i.e use a local directory as a “docs” resource, do a one-time upload/index from client→server, then retain it server-side as a re-usable doc index?