I’ve tried to add the following docs, and it failed for both sites - only reading one page for each:
https://reaflow.dev - maybe because it uses old school hash based routing
https://reactflow.dev - not sure why it failed
It should also be noted that realistically, React Flow has 3 separate and relevant entry points:
- /learn
- /examples
- /api-reference
So, it really should be possible to add all pages under those 3 URLs into @reactflow.
More generally, this would align with the request (from someone) to let the user add further links under a given docs handle themselves, if Cursor fails to find them.
–
Additionally, from my experience trying to just download React Flow’s docs by hand (awful)… They have sub-tabs for different code files (js,css,html, etc.), and only one is visible by default.
So, when crawling, you may have to activate those tabs, to grab the file contents under them.
^ I’m sharing that as an edge case to be aware of when crawling a docs website.
–
IMO, one intermediate solution for the handling of docs would be to just have all the crawled docs go into a .docs folder for Cursor (maybe under that project’s folder).
And then if the user wants to just drop in more files, they should be able to.
That may be worth having, even if the crawler is improved… In part because the user might have relevant docs of their own (from their own company) w.r.t a given library.