I have lots of markdown files (~ 30,000) that i’m trying to index with cursor. When building the index, it seems to work fine until it gets to roughly halfway. At this point, it just seems to get stuck. Below is what Cursor index has looked like for > 30 minutes:
The largest markdown file I have is 2.5 Mb, only ~10 of the 30k are over 1Mb.
I’ve seen elsewhere in the forum there is a 100k file limit, so I don’t think i’m hitting the file limit. I don’t see any error message in Cursor, it’s acting like it’s still syncing but nothing is happening.
Is this a known issue? is there somewhere I can find logs to see what’s going wrong?
I’ve tried deleting the index → restart Cursor → rebuild index, and i get the same results
Cursor details:
Version: 0.44.11
VSCode Version: 1.93.1
Commit: fe574d0820377383143b2ea26aa6ae28b3425220
Date: 2025-01-03T07:59:06.361Z
Electron: 30.5.1
Chromium: 124.0.6367.243
Node.js: 20.16.0
V8: 12.4.254.20-electron.0
OS: Darwin arm64 24.3.0
Hey, could you take a screenshot of the output in the Cursor View > Output menu and select Cursor Indexing & Retrieval from the dropdown menu?
Hello ! The issue is persisting.
Hey, I’ve thrown to the team to see what is going on here!
Hopefully will have a response for you soon.
My log file is showing a bit differently than the one above. Lots of waiting on semaphores, then it seems to give up Indexing job successfully done or aborted. and restart trying to index 30061 files all over again. It’s on it’s 6th retry right now, still showing 15.5%, logs still showing 30061 to go.
2025-01-23 12:47:46.308 [info] Waiting on semaphore to be released 20
2025-01-23 12:47:46.508 [info] Waiting on semaphore to be released 20
2025-01-23 12:47:46.709 [info] Waiting on semaphore to be released 20
2025-01-23 12:47:46.910 [info] Waiting on semaphore to be released 20
2025-01-23 12:47:47.116 [info] Waiting on semaphore to be released 16
2025-01-23 12:47:47.321 [info] Waiting on semaphore to be released 13
2025-01-23 12:47:47.521 [info] Waiting on semaphore to be released 8
2025-01-23 12:47:47.763 [info] setting numJobsToGo to 30230
2025-01-23 12:47:47.767 [info] [startSync]: numJobs: 30230
2025-01-23 12:48:48.808 [warning] Retrying fast update file (delete) with timeout 20000. Error: [deadline_exceeded] the operation timed out
2025-01-23 12:48:51.486 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:49:51.060 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:49:52.938 [warning] Retrying fast update file (delete) with timeout 20000. Error: [deadline_exceeded] the operation timed out
2025-01-23 12:50:57.933 [warning] Retrying fast update file (delete) with timeout 20000. Error: [deadline_exceeded] the operation timed out
2025-01-23 12:50:58.181 [info] Uploading 30230 files.
2025-01-23 12:51:04.963 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:51:05.102 [info] Completed 50 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:06.978 [info] Completed 100 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:09.314 [info] Completed 150 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:10.867 [info] Completed 200 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:12.483 [info] Completed 250 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:14.242 [info] Completed 300 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:15.584 [info] Completed 350 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:18.592 [info] Completed 400 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:19.840 [info] Completed 450 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:20.927 [info] Completed 500 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:22.116 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:51:22.759 [info] Completed 550 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:24.369 [info] Completed 600 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:25.915 [info] Completed 650 jobs for absoluteDirectoryPath: undefined
2025-01-23 12:51:27.333 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:51:27.337 [info] InternalRepoInfo constructor with preferredEmbeddingModel 0
2025-01-23 12:51:27.338 [info] Indexing job successfully done or aborted.
2025-01-23 12:51:27.344 [info] Creating merkle client.
2025-01-23 12:51:27.344 [info] Done creating merkle client.
2025-01-23 12:51:27.344 [info] Doing a startup handshake.
2025-01-23 12:51:49.117 [info] Finished initializing merkle tree in 21764.209875 ms.
2025-01-23 12:51:49.130 [info] Doing the initial handshake with hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 12:51:49.136 [info] Handshake start
2025-01-23 12:51:49.297 [info] Handshake timing: 162.03879200003576
2025-01-23 12:51:49.297 [info] Handshake result: {"status":"STATUS_OUT_OF_SYNC","repoName":"dd155abf-4662-4461-9ef3-b36b01ae4c71"}
2025-01-23 12:51:49.297 [info] Starting fast remote sync.
2025-01-23 12:51:49.307 [info] Total num embeddable files: 34945
2025-01-23 12:51:49.308 [info] Root hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 12:51:49.308 [info] In the out of sync case.
2025-01-23 12:51:49.308 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":".","hash":"467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03"}
2025-01-23 12:51:49.308 [info] Waiting on semaphore to be released 1
2025-01-23 12:51:49.513 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube","hash":"0c6506ed8cb94c546085196809a5a7c0ec6f9c9f276fa203be68cab5829998b7"}
2025-01-23 12:51:49.514 [info] Waiting on semaphore to be released 1
2025-01-23 12:51:50.023 [info] Waiting on semaphore to be released 1
2025-01-23 12:51:50.257 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0cXBLPYOzv4.localized","hash":"9cb2624ad027c5784d7b8574ff3ec016d2c3c6c344f909df453b5d747b898cad"}
2025-01-23 12:51:50.258 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0byLiNzDrdE.localized","hash":"9151398c018e6e31a4669c998f9b4f33f83b06a2e1c93df9136737e35db1f981"}
// snip
2025-01-23 12:51:50.277 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/--6eyLO78CY.localized","hash":"de47b24a815a92891622b416a72632d43db51ae549f96522539d5125a59e366d"}
2025-01-23 12:51:50.277 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/-7ghjFTWMtc.localized","hash":"917e01d6d0305ddac8cd13c3ef5a9d08ba59a750307966e32fdcbaa3b43eed01"}
2025-01-23 12:51:50.277 [info] Waiting on semaphore to be released 20
2025-01-23 12:51:50.457 [info] Waiting on semaphore to be released 20
2025-01-23 12:51:50.657 [info] Waiting on semaphore to be released 20
// snip
2025-01-23 12:51:57.395 [info] Waiting on semaphore to be released 20
2025-01-23 12:51:57.595 [info] Waiting on semaphore to be released 20
2025-01-23 12:51:57.671 [info] Creating Indexing Repo client: https://repo42.cursor.sh
2025-01-23 12:51:57.674 [info] InternalRepoInfo constructor with preferredEmbeddingModel 0
2025-01-23 12:51:57.691 [info] Creating merkle client.
2025-01-23 12:51:57.691 [info] Done creating merkle client.
2025-01-23 12:51:57.691 [info] Doing a startup handshake.
2025-01-23 12:51:57.800 [info] Indexing job successfully done or aborted.
2025-01-23 12:52:17.221 [info] Finished initializing merkle tree in 19503.74558400002 ms.
2025-01-23 12:52:17.231 [info] Doing the initial handshake with hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 12:52:17.237 [info] Handshake start
2025-01-23 12:52:17.527 [info] Handshake timing: 291.14675000001444
2025-01-23 12:52:17.527 [info] Handshake result: {"status":"STATUS_OUT_OF_SYNC","repoName":"dd155abf-4662-4461-9ef3-b36b01ae4c71"}
2025-01-23 12:52:17.528 [info] Starting fast remote sync.
2025-01-23 12:52:17.533 [info] Total num embeddable files: 34945
2025-01-23 12:52:17.535 [info] Root hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 12:52:17.535 [info] In the out of sync case.
2025-01-23 12:52:17.535 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":".","hash":"467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03"}
2025-01-23 12:52:17.535 [info] Waiting on semaphore to be released 1
2025-01-23 12:52:17.740 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube","hash":"0c6506ed8cb94c546085196809a5a7c0ec6f9c9f276fa203be68cab5829998b7"}
2025-01-23 12:52:17.741 [info] Waiting on semaphore to be released 1
2025-01-23 12:52:17.939 [info] Waiting on semaphore to be released 1
2025-01-23 12:52:18.309 [info] Waiting on semaphore to be released 1
2025-01-23 12:52:18.537 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0cXBLPYOzv4.localized","hash":"9cb2624ad027c5784d7b8574ff3ec016d2c3c6c344f909df453b5d747b898cad"}
2025-01-23 12:52:18.538 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0byLiNzDrdE.localized","hash":"9151398c018e6e31a4669c998f9b4f33f83b06a2e1c93df9136737e35db1f981"}
// snip
2025-01-23 12:52:18.560 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/--6eyLO78CY.localized","hash":"de47b24a815a92891622b416a72632d43db51ae549f96522539d5125a59e366d"}
2025-01-23 12:52:18.560 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/-7ghjFTWMtc.localized","hash":"917e01d6d0305ddac8cd13c3ef5a9d08ba59a750307966e32fdcbaa3b43eed01"}
2025-01-23 12:52:18.560 [info] Waiting on semaphore to be released 20
2025-01-23 12:52:18.742 [info] Waiting on semaphore to be released 20
2025-01-23 12:52:18.941 [info] Waiting on semaphore to be released 20
// snip
2025-01-23 12:52:42.327 [info] Waiting on semaphore to be released 20
2025-01-23 12:52:42.528 [info] Waiting on semaphore to be released 18
2025-01-23 12:52:42.828 [info] Waiting on semaphore to be released 12
2025-01-23 12:52:43.031 [info] Waiting on semaphore to be released 8
2025-01-23 12:54:11.250 [info] setting numJobsToGo to 30061
2025-01-23 12:54:11.250 [info] [startSync]: numJobs: 30061
2025-01-23 13:01:57.675 [info] Creating merkle client.
2025-01-23 13:01:57.675 [info] Done creating merkle client.
2025-01-23 13:01:57.675 [info] Doing a startup handshake.
2025-01-23 13:02:18.542 [info] Finished initializing merkle tree in 20857.11020799994 ms.
2025-01-23 13:02:18.553 [info] Doing the initial handshake with hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 13:02:18.559 [info] Handshake start
2025-01-23 13:02:18.746 [info] Handshake timing: 186.70879200007766
2025-01-23 13:02:18.746 [info] Handshake result: {"status":"STATUS_OUT_OF_SYNC","repoName":"dd155abf-4662-4461-9ef3-b36b01ae4c71"}
2025-01-23 13:02:18.746 [info] Starting fast remote sync.
2025-01-23 13:02:18.752 [info] Total num embeddable files: 34945
2025-01-23 13:02:18.753 [info] Root hash: 467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03
2025-01-23 13:02:18.753 [info] In the out of sync case.
2025-01-23 13:02:18.753 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":".","hash":"467d91e6fd8b8f328a380d908186555ab6030970165edc5d619ff0fb84cd3a03"}
2025-01-23 13:02:18.753 [info] Waiting on semaphore to be released 1
2025-01-23 13:02:18.958 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube","hash":"0c6506ed8cb94c546085196809a5a7c0ec6f9c9f276fa203be68cab5829998b7"}
2025-01-23 13:02:18.959 [info] Waiting on semaphore to be released 1
2025-01-23 13:02:19.162 [info] Waiting on semaphore to be released 1
2025-01-23 13:02:19.460 [info] Waiting on semaphore to be released 1
2025-01-23 13:02:19.949 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0cXBLPYOzv4.localized","hash":"9cb2624ad027c5784d7b8574ff3ec016d2c3c6c344f909df453b5d747b898cad"}
2025-01-23 13:02:19.950 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/0byLiNzDrdE.localized","hash":"9151398c018e6e31a4669c998f9b4f33f83b06a2e1c93df9136737e35db1f981"}
// snip
2025-01-23 13:02:19.970 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/--6eyLO78CY.localized","hash":"de47b24a815a92891622b416a72632d43db51ae549f96522539d5125a59e366d"}
2025-01-23 13:02:19.970 [info] [startSync]: ----------------------
syncing point nextSubtree{"relativePath":"./YouTube/-7ghjFTWMtc.localized","hash":"917e01d6d0305ddac8cd13c3ef5a9d08ba59a750307966e32fdcbaa3b43eed01"}
2025-01-23 13:02:19.970 [info] Waiting on semaphore to be released 20
2025-01-23 13:02:20.149 [info] Waiting on semaphore to be released 20
2025-01-23 13:02:20.349 [info] Waiting on semaphore to be released 20
// snip
2025-01-23 13:02:44.969 [info] Waiting on semaphore to be released 20
2025-01-23 13:02:45.180 [info] Waiting on semaphore to be released 19
2025-01-23 13:02:45.399 [info] Waiting on semaphore to be released 15
2025-01-23 13:02:45.608 [info] Waiting on semaphore to be released 13
2025-01-23 13:02:45.808 [info] Waiting on semaphore to be released 4
2025-01-23 13:04:15.570 [info] setting numJobsToGo to 30061
2025-01-23 13:04:15.570 [info] [startSync]: numJobs: 30061
My log file looks a bit different than the one posted above.
Here’s the middle of the file, showing 30061 files to sync:
and here’s the end of the log, where it appears to give up with the same number of files still to go:
In this attempt, the sync is stuck at 15.5%:
Hey, does the same thing occur if you delete the index and restart it?
Also, some users report that renaming the folder your code is in helps, as this is treated as a fresh codebase by Cursor, but this would cause your chats and composers to disappear for the project if you choose to rename it.
Yes, I’ve tried pausing and restarting, and i’ve also tried deleting the index and retrying fresh → the index gets stuck in both cases.
I have 3 folders i’m indexing, they all have essentially the same structure - thousands of entries with one subdirectory per entry containing image/json/markdown/webloc files. My .cursorignore removes all except the markdown files from indexing, and when i count them, the three directories have 30228, 2302, and 2435 markdown files respectively.
If i add the directory with 30228 files to my .gitignore, the index completes just fine. however, when that directory is not ignored, it consistently gets stuck indexing. I wonder if it’s a specific file in that folder it’s getting stuck on? I’ll try and do some more experiments this evening to narrow down why and when it gets stuck indexing that directory, but those are my clues so far.
So here’s an interesting data point. If i add a ~5k of the 30228 at a time, and reindex with each batch, i can get it to complete the full index. However, if i rebuild the index from scratch with all ~35k items, then it’ll get stuck at the ~50% ish mark.
Thanks for investigating when this issue occurs!
I’ll pass this on to the team to figure out whats going on behind the scenes here.
