I’m impressed by how the agent can run console applications and iteratively correct mistakes using console output. I’d love for it to also interact with web pages generated by frameworks like NextJS.
Since it can’t directly see my browser, could I use a workaround? For example, could NextJS generate simplified markup that is outputted in the console, along with commands as console inputs, so the agent could interact with them by seeing and issuing commands?
Has anyone else explored or considered giving agents access to generated web pages?