Bulk Adding Docs

Clicking the ‘Add new doc’ button in the Features panel of Cursor Settings allows you to add a new docs URL to be indexed. If find that sometimes Cursor will spider multiple pages from this URL, but other times I find it does not.

For example, I’ll post the to URL documentation for an API and Cursor will index the first few pages, but not all pages in the documentation. I end up having to manually add each page in the documentation by clicking ‘Add new doc’ for every page.

Is there a way to bulk add multiple pages? Perhaps a config file where these pages can be added instead of having to press ‘Add new doc’ and use the UI for every page?

1 Like

It’s not possible at the moment. Thanks for the feature request!

Out of interest, how many docs do you want to add? We’ve pre-indexed a lot of common docs so you shouldn’t have to add any popular/standard packages.

Also, which docs are you having trouble scraping?

Thanks so much for your quick reply …

As an example, this week I am building a business analytics dashboard for a new client. I need to integrate five different third-party APIs into my code. I’d like to add the documentation for all five API services to Cursor.

If the documentation is served in a way that Cursor cannot easily spider, I have to enter each page of of the five docs manually. Doing this from the modal pop launched from the Settings panel is not ideal.

A UI that allows me to more easily add multiple links at once would be ideal. I am envisioning an interface where I can add infinite rows before submitting. And I can also upload a CSV or paste the CSV into the first row of the UI and have it add all the rows needed for the data I’ve pasted. This would allow me to add pages much faster.

The docs for Airtable Extensions (Airtable Blocks SDK) is an example of documentation where I found Cursor unable to spider well and myself having to manually add several links. Another set of docs that Cursor could not spider are https://flowhub.stoplight.io/docs/public-developer-portal/4b402d5ab3edd-welcome.

I find the Docs support to be one of the most powerful features of the software, so I’d love to be able to manage the indexing more easily.

Had trouble with LangGraph (Python), although after trying later the same day it seemed to work.
This link: Home

1 Like

Sorry to resurrect this post, but this is still an issue for me. Here’s an example of a docs site where I have to enter every page by hand and it’s unable to spider the pages despite a nav appearing on each page:

Would be nice to fix the spidering issue and/or provide a way to add multiple pages from a single interface.

This is weird … one of the pages I added from the docs linked above picked up 39 pages which is perhaps the entire documentation:

But oddly, I don’t see any links on that page that might allow the full site to be spidered.

Would love to better understand how indexing works in hopes I can figure out workarounds to not have to add each page when this issue pops up.

I am running into this issue as well with the AvantLink API docs. The nav is in a simple UL element and it doesn’t properly crawl them.

The problem with this documentation is that the URL doesn’t actually change between the pages of the documentation. It uses a parameter for a PHP-based website.

For example:
https://classic.avantlink.com/api.php?help=1&module=AdSearch
and
https://classic.avantlink.com/api.php?help=1&module=ConfigureDatafeed
both have
https://classic.avantlink.com/api.php?
as their URL, which make the AI think it’s the same page!

This is a bit of an edge case, but I appreciate the report as we might be able to fix this in the future!

1 Like