Quick update - I managed to make it fast by clearing all the possible cache. This will also fully remove your settings/account, but it works - Cursor seems snappy now, like it should be.
I had 16GB to remove.
Run with caution:
rm -rf ~/Library/Application\ Support/Cursor/{Cache,CachedData,CachedExtensionVSIXs,GPUCache,Code\ Cache,Service\ Worker,DawnGraphiteCache,DawnWebGPUCache,logs,WebStorage} && \
rm -f ~/Library/Application\ Support/Cursor/User/globalStorage/state.vscdb* && \
rm -rf ~/Library/Application\ Support/Cursor/User/workspaceStorage && \
rm -rf ~/Library/Application\ Support/Cursor/User/History
Hopefully someone can tell me what to discard from this command, to not remove my settings next time when I want to clean.