Yesterday Anthropic released several new versions of its Claude AI foundation model. Most notably, and what’s garnering much buzz, is they’ve introduced a new feature called “computer use”. What this means is that Claude is actually able to control your computer - launch apps, click buttons, type text, etc. It’s in a very early experimental phase, but in my opinion (and many others in the AI field) this is truly the beginning of a new era.
Here is the official announcement:
Note BTW one of the companies listed in the announcement:
Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company have already begun to explore these possibilities, carrying out tasks that require dozens, and sometimes even hundreds, of steps to complete.
I decided to give it a spin. As an initial test, I took a simple example of the frequent forum request to be able to update existing Asana data from a spreadsheet/CSV.
Here is the demonstration:
(TLDR: it worked!)
Some notes:
-
Per Anthropic’s recommendation not to use it in a live environment yet, this example is running in a simple Docker container, using one of my test Asana workspaces.
-
Currently the “computer use” feature requires using the Claude API; you can’t just type a prompt into the regular Claude prompt window to get it to control your computer. (Not yet, anyway…) Also, to be clear, this is just an experiment on my part and is in no way any type of official announcement or support by Asana of this new feature.
-
Literally the only thing I did was paste in the prompt on the left and press [Enter]. My hands never touched the mouse or keyboard after that; everything else you see was done by Claude.
-
It’s running in a verbose mode where you can see on the left side every step that it takes. Basically how “computer use” works is that when you start, you tell it your screen height and width in pixels. From there, it takes a screenshot, calculates by pixel location where it needs to take the appropriate action, executes that action, then takes another screenshot and evaluates it to decide if the action it just took worked properly. It repeats this process for every step that it decides it needs to do.
-
What you see is only my second try; I only had to change one thing. On my first try, I didn’t have the last sentence of “To update a due date, click in the due date column for that task, blank out the existing due date first, then type in the new due date.” and Claude did not figure out how to change the due date (many users have trouble with this as well!) But you’ll see that once I added that extra prompt, it worked perfectly!
Again, per Anthropic’s guidance, this is not ready for production use yet. But it’s only a matter of time…
)
