How much longer till an code helper actively monitors what you are coding and what the results on the screen are (+ local machine access)?

Discussions
1

It seems the next step right?

Now we need to feed it what we are coding manually but since they can already understand what is happening on our screens (like sending a screen shot of an error to ChatGPT) then why not have an app that actively does that and can be aware of the context of the code AND the results of the code?

I can imagine an assistant in the IDE given constant feedback as you compile and run …

Maybe even give it total access to your local machine… For example the code can’t find a file then the assistant can see you did …/…/…something instead of …/…/something and can be aware of that also… of maybe npm configs… etc…

Is anyone aware of anything like this ?

2

It’s getting there.

For example Copilot++ looks at the code you have recently written to make its prediction.

You can quickly copy & paste terminal output to the chat or use the Auto Terminal Debug feature.

Interpreter Mode can run shell commands for you, create files etc.

3

I guess the main thing is some sort of constant awareness of your entire screen. .so simple example I had the other day with unity and chatGPT

I had a button that was not changing color after a certain amount of time… I used the code it gave me and I told it I need it to change color after 2 seconds…

If somehow the AI was aware of the running of the game and could translate what it saw to match up its understanding then that woudl be next level right?

4

Hi Matthew,

As of right now, we don’t have a plan to record the entire screen (one reason being privacy).

But a good hack here is just taking a screenshot of your button and feeding that to the cmdk prompt.

I find that really helpful for when I’m changing ui from somewhere else.

5

Yes. That is what I do now and it works pretty good… screen shot the console and screen and hope it understand the UI

Im fine with it recording my screen so im all for an app that can take it to that next level