Request refund please - feedback within

[leonstafford]

Hi there,

I was encouraged by a friend to use Cursor, due to the touted whole codebase inference. Signed up this morning and gave it a try with a project.

Unfortunately, I saw no difference, actually, less quality than simply GPT-4 with an uploaded codebase zip.

ie, you can see from my chat transcript for account with matching username, that there was no AI knowledge of main libraries being used within the project. It would hallucinate with simple, direct request to use methods from that library. The editor’s intellisense-type autocompletion was correct in its understanding of the library’s method, but the AI refused to analyse the library’s files to use the correct methods, instead, making up non-existant methods.

For the price and advertised whole codebase AI assistance, I was expecting more than what is achievable with GPT-4 alone and it fell short of expectations.

GPT-4 with a whole codebase zip uploaded doesn’t do much better, but at least, I can then upload a zip of the library in question and get it to analyse that and eventually suggest code based on existant methods in the library.

IMO, Cursor should be doing this, either with existant knowledge of public source vendor libraries based on a dependency file, or to simply scan all the files in the project, as it does for the intellisense-type autocompletion.

That there were minimal GPT calls showing after my operations in my account quota, it simply seems not to be sending enough context in the prompts or not handling the local knowledge of the codebase well in the AI integration.

If you see an obvious error in my understanding, happy to try and use the product again, else would like to request a refund as it’s not usable for my needs.

Cheers,

Leon

[debian3]

Maybe you should check in MORE (in the right sidebar) that you codebase is properly indexed

Then you can prompt @codebase in the chat.

[leonstafford]

Checked now and was fully synced and codebase is fairly small, so assume it was when I tested earlier.

Tried again with fresh chats, using variations:

• same prompt as before
• with @codebase
• with “Interpereter Mode (Beta)” enabled

Same issues, with hallucinations / making assumptions about non-existent methods instead of referring to the code/package source/package documentation I provided (and which the auto-complete within IDE is aware of.

[debian3]

For me it works really well. It works better than ChatGPT because of the preprompt they use that are automatically generated. I would try flushing the index and redoing it or reinstalling Cursor. Are you using GPT4?

But I’m sure someone from Cursor will help you in more details.

[leonstafford]

Was using GPT4 when encountered the issue. Are you explicitly getting responses including code from your vendor files, not already in GPT-4’s trained model?

That is the crux of my issue and perhaps it’s due to this particular vendor’s files not being as popular or old as other ones in GPT-4’s learned model. ie, troubleshooting further today, I get similar issues with any GPT-4 backed tool, unless I forcefully spoon feed it the vendor files.

For Cursor/VSCode, this is especially annoying, as it otherwise has access to the whole project, just seems to be being frugal with token usage not to scan deeper.

I’m glad it works well for your particular use case, but don’t see this working adequately for me.

Cursor team can look at my logs/prompt history and hopefully infer something, but I’m not hopeful.

[leonstafford]

(similar issues with VSCode’s @Workspace, so I simply think we’re not there yet with useful whole codebase/project integration with GPT-4 in any of these tools, if we were, they should probably be costing more in the amount of tokens they’d require to fine-tune/send enough data in prompting)

[npour]

Sorry to hear Leon.

I’ve refunded your charge. Kept your current plan active till Apr 8th too.

As for the product, we do not yet automatically inject libraries into your prompt. You could use the @doc feature to achieve this by tagging the libraries you use.

You could also add other documentation that you think are useful using @doc. Cursor will automatically index the documents and make them easily searchable for you.

Hope this helps

[leonstafford]

Thanks @npour,

The @docs looks like a promising feature, but unfortunately, even after feeding it this package’s online documentation and referencing it, it was using non-existant methods.

What did help, though, was to remove the Python venv dir from my .gitignore and re-index in Cursor. This stopped it making up methods from the package and with a bit more guidance, referred to the actual method I was aiming for. It started making things up again with further questioning, but looks promising.

I think this was the crux of my problem and may be an area of improvement.

ie, most projects will have their dependencies in their .gitignore, such as node_modules, venv, etc. Cursor ignores these from indexing, maybe to reduce cost of sending that extra context to OpenAI?

As a new user, I’d want Cursor to know my codebase intimately. Even if adding a @libraries or such tag, one would need to know to include that, whereas, I expect @Codebase to mean the whole codebase. Similarly, I don’t think force including via the .cursorignore would be a good DX, either, it should JustWork™.

I hope this feedback helps.

I may not be the target demographic, as I’m happiest in a minimal Vim + tmux setup in OpenBSD, but last year, I was motivated enough with CoPilot functionality to spend some time in VSCode and for times when I want to have whole codebase inference, I’d happily pay more than US$20/month for something powerful.

As it is now, I feel I can’t justify the extra guidance / troubleshooting with Cursor/VSCode vs spending the same time to either manually solve or explicitly drive ChatGPT with GPT-4, to get whole codebase functionality. At the other end of the cost/complexity scale would be fine-tuning an LLM on my codebase and writing my own wrapper. So, somewhere in the middle of that, is where I’d pay for a solid whole codebase AI in Cursor.

Happy to help test such functionalities in future if things go that way!

Cheers,

Leon

[debian3]

@leonstafford

Was using GPT4 when encountered the issue. Are you explicitly getting responses including code from your vendor files, not already in GPT-4’s trained model?

No, but when I’m working on certain library I will feed it the library documentation with @Docs and it works quite good.

ChatGPT the context window is larger (128k) vs Cursor (or Copilot) at around 10k. That’s probably what you are noticing here. Cost increase dramatically with larger context window. If you using Gemini 1M token, and you end up over $1 per interaction with the chat…

You can do a search here about the context window size, there was debate over this already and there is other consideration as well with the current model limitation when you increase the context window size.

[HappyQuokka]

how is it possible that gpt4 reads all zip, where do you use zip uploads, in the chatgpt plus subscription? last time I used it, the thing forgets previous paragraphs of conversation rather quickly how come it analyzes codebase and libraries for you, which language do you work with? (most likely it just guesses because of matching function names and its heuristics kind of, you can’t make it ‘really scan and remember’ more than 20-30kb of text (16k tokens) or let’s say 40-50kb of text (with 32k tokens model), what am I missing?

But for me the embeddings in Cursor work well so far, even with the gpt3.5 on top of them, not perfect but I can see it does find the correct info in the docs which I gave it (roughly 500 pages).

[HappyQuokka]

fine tuning a local model like DeepSeek coder or StarCoder2, seems like the next step to try, for those who can’t make the Cursor embeddings approach work well for their use case. the fine tuning will need a dataset but you can make an existing local model to go one by one through all functions and classes in the codebase, literally file by file, and sort them into a “question/answer/description” format that’s used for fine-tuning.

I’m surprised people didn’t make a system like this yet, which will first scan the existing code base to make it as ‘data set’, and then fine-tune something, and you get a model as a result that’s fully familiar with a project and all its functions and variables and documentation. then all updates to the codebase go into the embeddings and work perfect with Cursor let’s say.

That’ll be a killer feature, and the fine-tuning should cost not more than 20-30$ even on Azure/AWS hardware (without mentioning cheaper gpu options which are a bit more complex to automate for such use-case).

[leonstafford]

Chat GPT with Plus plan has built-in file uploads now (where it used to be selectively rolled out, under a different name). The same issue with making up non-existant methods happens with the chat functionality of Cursor, VSCode or ChatGPT. I did notice it was able to code using the correct methods from comments within VSCode and GH Codespace, at least, but their chat was still making the same mistakes. I assume from the differences between Chat vs Complete APIs?

I can load the libraries into the ChatGPT in addition to my repo’s files (or zip up the whole project, including depency files) and make it work, that’s where the advantage is for that at the moment.

[leonstafford]

Yes, if I were desperate to get whole codebase AI working well, I’d look to train my own model/fine-tune StarCoder. As it is, I’ll go back to focused work in basic Vim + tmux and revisit the IDE space once these things have advanced to where the Chat has more knowledge of the code.

It’s frustrating to know what’s possible, but that we’re constrained mostly by resources.

[myriam]

Such a fascinating discussion, thanks for putting out the words all, well laid out.

Glad I came to the forum a month later to witness it, as I had these thoughts in the back of my mind for a while. I’m gonna give a try to Cursor Pro now and find out by myself how it indexes my projects and responds, considering I also have GPT4 Plus and Copilot, but none of them are capable of maintaining context of my full code base.

Hopefully the Cursor flow will be smooth and accurate.