Failed to index documentation - OpenAI Cookbook

Hello, I am trying to index the following pages using Cursor, but am running into an error when attempting to index this page specifically: Summarizing Long Documents | OpenAI Cookbook

The first custom page I have indexed is youtube-transcript-api · PyPI, which likely is unnecessary since Cursor likely has pypi indexed already, but I noticed that every version of the package was indexed when I added the page. I only care about using the latest version. I wonder if there is possibly a limit on the number of pages that can be indexed, and indexing all the versions of this package may have used up the limit. While there could be a limit, I think that isn’t the issue since after deleting the indexes for the YT-transcript package, the indexing still failed.

I’m curious what strategies folks here have found successful. Do you curate a set of docs in a folder of txt files and stick that in your root dir so Cursor indexes it with the rest of your code instead of scraping the web for it? Also does anyone know if there is a limit? Btw, I saw the official OAI docs indexed, but not the cookbook. The cookbook would be great too.

1 Like

Here is the answer: Can Cursor read and understand a Git repo? - #2 by Jakob