Hey, do you have some examples of what documents aren’t correctly indexing here?
While the crawler does occasionally fail due to network issues, most of the time it works perfectly fine!
These docs fail: effect - effect
It doesn’t crawl any of the links. It’s really unfortunate because much of my app is based on effect-ts so cursor doesn’t understand a lot of it.
In addition to that any github repo URL might be seen as a failure case – it won’t crawl the code, it seems to pick a few random links. I tried using https://uithub.com but that also fails maybe because the resulting page of concatenated sources is above some size threshold.
For some websites you may have to be more specific about the URLs that you enter for the crawler to use.
I managed to get your example working with the following settings as you can see in the second screenshot.
Ah yeah I did eventually get it to crawl by playing with the link. The github case is still a problem however. If I want to refer to the source code of a library like effect I have to clone a local copy of it since the docs function can’t properly crawl source trees on github.