I think I have identified what is causing the issue: Apparently Cursor only indexes / accesses only the first folder when working with code-workspaces. See bug report here.
When opening a normal folder, everything works as expected. Both with indexing enabled and disabled (when disabled, Cursor writes out a plan and queries the files live).
But when using a workspace, Cursor cannot access the files. Independent of indexing.
It would be great to allow Cursor access to all workspace folders or at least somehow indicate that not all folder are visible to Cursor.