Clicking the ‘Add new doc’ button in the Features panel of Cursor Settings allows you to add a new docs URL to be indexed. If find that sometimes Cursor will spider multiple pages from this URL, but other times I find it does not.
For example, I’ll post the to URL documentation for an API and Cursor will index the first few pages, but not all pages in the documentation. I end up having to manually add each page in the documentation by clicking ‘Add new doc’ for every page.
Is there a way to bulk add multiple pages? Perhaps a config file where these pages can be added instead of having to press ‘Add new doc’ and use the UI for every page?
As an example, this week I am building a business analytics dashboard for a new client. I need to integrate five different third-party APIs into my code. I’d like to add the documentation for all five API services to Cursor.
If the documentation is served in a way that Cursor cannot easily spider, I have to enter each page of of the five docs manually. Doing this from the modal pop launched from the Settings panel is not ideal.
A UI that allows me to more easily add multiple links at once would be ideal. I am envisioning an interface where I can add infinite rows before submitting. And I can also upload a CSV or paste the CSV into the first row of the UI and have it add all the rows needed for the data I’ve pasted. This would allow me to add pages much faster.
Sorry to resurrect this post, but this is still an issue for me. Here’s an example of a docs site where I have to enter every page by hand and it’s unable to spider the pages despite a nav appearing on each page:
Would be nice to fix the spidering issue and/or provide a way to add multiple pages from a single interface.
The problem with this documentation is that the URL doesn’t actually change between the pages of the documentation. It uses a parameter for a PHP-based website.
For example: https://classic.avantlink.com/api.php?help=1&module=AdSearch
and https://classic.avantlink.com/api.php?help=1&module=ConfigureDatafeed
both have https://classic.avantlink.com/api.php?
as their URL, which make the AI think it’s the same page!
This is a bit of an edge case, but I appreciate the report as we might be able to fix this in the future!