Custom docs are potentially a great feature but they are really limited by the fact that we can only use one URL as the entry point! The crawler constantly misses sites.
Here is an example: I added the Pinecone documentation using this as the URL (which is the starting page for their DOCS): Pinecone Documentation - Pinecone Docs
The crawler gets 14 pages but misses many. The 14 pages include completely empty pages and misses many important one like this one:
- Query data - Pinecone Docs
- And many more… (I can only include two links as a new forum user)