Anthropic teaches its AI model Claude how to use a computer. The company demonstrates how the model controls the cursor, searches the Internet for places to visit near the user's home, and even adds a route to the desktop calendar.
The developers say that instead of creating dozens of different AI tools for each individual task, they decided to teach Claude "computer skills" so that it could do it on its own. The feature is currently in "public beta" for developers.
Claude can control the computer by taking many screenshots and sending them for processing. The model can learn what is on the screen, including the distance between the cursor and the button to be pressed. It also returns commands to continue the task.
Anthropic says this feature could be used for repetitive tasks or open-ended research. Claude can be instructed to review a user's social media pages and delete old posts. Or, you can ask the model to act out a hectic activity in front of the tracker.
Anthropic says that during testing, Claude even took a break from writing code and searched for images of nature on Google.
Even while recording these demos, we encountered some amusing moments. In one, Claude accidentally stopped a long-running screen recording, causing all footage to be lost.
— Anthropic (@AnthropicAI) October 22, 2024
Later, Claude took a break from our coding demo and began to peruse photos of Yellowstone National Park. pic.twitter.com/r6Lrx6XPxZ
Also, some actions familiar to humans, such as scrolling the screen, dragging objects, or zooming, are difficult for the model to perform. During one of the tests, Claude stopped recording the screen and messed up the test recording. Even the developers themselves warn that the new feature should be tested with "low-risk tasks."