Getting began with laptop use in Amazon Bedrock Brokers

Pc use is a breakthrough functionality from Anthropic that permits basis fashions (FMs) to visually understand and interpret digital interfaces. This functionality allows Anthropic’s Claude fashions to establish what’s on a display screen, perceive the context of UI parts, and acknowledge actions that must be carried out reminiscent of clicking buttons, typing textual content, scrolling, and navigating between purposes. Nevertheless, the mannequin itself doesn’t execute these actions—it requires an orchestration layer to securely implement the supported actions.

Immediately, we’re saying laptop use assist inside Amazon Bedrock Brokers utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude Sonnet 3.7 fashions on Amazon Bedrock. This integration brings Anthropic’s visible notion capabilities as a managed device inside Amazon Bedrock Brokers, offering you with a safe, traceable, and managed approach to implement laptop use automation in your workflows.

Organizations throughout industries wrestle with automating repetitive duties that span a number of purposes and methods of file. Whether or not processing invoices, updating buyer information, or managing human useful resource (HR) paperwork, these workflows usually require workers to manually switch data between completely different methods – a course of that’s time-consuming, error-prone, and tough to scale.

Conventional automation approaches require customized API integrations for every software, creating important improvement overhead. Pc use capabilities change this paradigm by permitting machines to understand current interfaces simply as people.

On this put up, we create a pc use agent demo that gives the crucial orchestration layer that transforms laptop use from a notion functionality into actionable automation. With out this orchestration layer, laptop use would solely establish potential actions with out executing them. The pc use agent demo powered by Amazon Bedrock Brokers offers the next advantages:

• Safe execution setting – Execution of laptop use instruments in a sandbox setting with restricted entry to the AWS ecosystem and the online. It’s essential to notice that presently Amazon Bedrock Agent doesn’t present a sandbox setting
• Complete logging – Skill to trace every motion and interplay for auditing and debugging
• Detailed tracing capabilities – Visibility into every step of the automated workflow
• Simplified testing and experimentation – Diminished danger when working with this experimental functionality via managed controls
• Seamless orchestration – Coordination of advanced workflows throughout a number of methods with out customized code

This integration combines Anthropic’s perceptual understanding of digital interfaces with the orchestration capabilities of Amazon Bedrock Brokers, creating a robust agent for automating advanced workflows throughout purposes. Moderately than construct customized integrations for every system, builders can now create brokers that understand and work together with current interfaces in a managed, safe manner.

With laptop use, Amazon Bedrock Brokers can automate duties via fundamental GUI actions and built-in Linux instructions. For instance, your agent might take screenshots, create and edit textual content recordsdata, and run built-in Linux instructions. Utilizing Amazon Bedrock Brokers and suitable Anthropic’s Claude fashions, you need to use the next motion teams:

• Pc device – Permits interactions with consumer interfaces (clicking, typing, scrolling)
• Textual content editor device – Gives capabilities to edit and manipulate recordsdata
• Bash – Permits execution of built-in Linux instructions

Answer overview

An instance laptop use workflow consists of the next steps:

1. Create an Amazon Bedrock agent and use pure language to explain what the agent ought to do and the way it ought to work together with customers, for instance: “You’re laptop use agent able to utilizing Firefox net browser for net search.”
2. Add the Amazon Bedrock Brokers supported laptop use motion teams to your agent utilizing CreateAgentActionGroup API.
3. Invoke the agent with a consumer question that requires laptop use instruments, for instance, “What’s Amazon Bedrock, are you able to search the online?”
4. The Amazon Bedrock agent makes use of the device definitions at its disposal and decides to make use of the pc motion group to click on a screenshot of the setting. Utilizing the return management functionality of Amazon Bedrock Brokers, the agent the responds with the device or instruments that it needs to execute. The return management functionality is required for utilizing laptop use with Amazon Bedrock Brokers.
5. The workflow parses the agent response and executes the device returned in a sandbox setting. The output is given again to the Amazon Bedrock agent for additional processing.
6. The Amazon Bedrock agent continues to reply with instruments at its disposal till the duty is full.

You possibly can recreate this instance within the us-west-2 AWS Area with the AWS Cloud Growth Equipment (AWS CDK) by following the directions within the GitHub repository. This demo deploys a containerized software utilizing AWS Fargate throughout two Availability Zones within the us-west-2 Area. The infrastructure operates inside a digital non-public cloud (VPC) containing public subnets in every Availability Zone, with an web gateway offering exterior connectivity. The structure is complemented by important supporting providers, together with AWS Key Administration Service (AWS KMS) for safety and Amazon CloudWatch for monitoring, making a resilient, serverless container setting that alleviates the necessity to handle underlying infrastructure whereas sustaining strong safety and excessive availability.

The next diagram illustrates the answer structure.

On the core of our answer are two Fargate containers managed via Amazon Elastic Container Service (Amazon ECS), every protected by its personal safety group. The primary is our orchestration container, which not solely handles the communication between Amazon Bedrock Brokers and finish customers, but in addition orchestrates the workflow that allows device execution. The second is the environment container, which serves as a safe sandbox the place the Amazon Bedrock agent can safely run its laptop use instruments. The setting container has restricted entry to the remainder of the ecosystem and the web. We make the most of service discovery to attach Amazon ECS providers with DNS names.

The orchestration container contains the next parts:

• Streamlit UI – The Streamlit UI that facilitates interplay between the tip consumer and laptop use agent
• Return management loop – The workflow liable for parsing the instruments that the agent needs to execute and returning the output of those instruments

The setting container contains the next parts:

• UI and pre-installed purposes – A light-weight UI and pre-installed Linux purposes like Firefox that can be utilized to finish the consumer’s duties
• Software implementation – Code that may execute laptop use device within the setting like “screenshot” or “double-click”
• Quart (RESTful) JSON API – An orchestration container that makes use of Quart to execute instruments in a sandbox setting

The next diagram illustrates these parts.

Stipulations

1. AWS Command Line Interface (CLI), observe directions right here. Make certain to setup credentials, observe directions right here.
2. Require Python 3.11 or later.
3. Require Node.js 14.15.0 or later.
4. AWS CDK CLI, observe directions right here.
5. Allow mannequin entry for Anthropic’s Claude Sonnet 3.5 V2 and for Anthropic’s Claude Sonnet 3.7.
6. Boto3 model >= 1.37.10.

Create an Amazon Bedrock agent with laptop use

You need to use the next code pattern to create a easy Amazon Bedrock agent with laptop, bash, and textual content editor motion teams. It’s essential to supply a suitable motion group signature when utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude 3.7 Sonnet as highlighted right here.

import boto3
import time

# Step 1: Create the bedrock agent consumer

bedrock_agent = boto3.consumer("bedrock-agent", region_name="us-west-2")

# Step 2: Create an agent

create_agent_response = create_agent_response = bedrock_agent.create_agent(
        agentResourceRoleArn=agent_role_arn, # Amazon Bedrock Agent execution function
        agentName="computeruse",
        description="""Instance agent for laptop use. 
				This agent ought to solely function on 
				Sandbox environments with restricted privileges.""",
        foundationModel="us.anthropic.claude-3-7-sonnet-20250219-v1:0",      
		instruction="""You're laptop use agent able to utilizing Firefox 
                 net browser for net search.""",
)

time.sleep(30) # await agent to be created

# Step 3.1: Create and connect laptop motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="ComputerActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Pc",
    parentActionGroupSignatureParams={
        "kind": "computer_20250124",
        "display_height_px": "768",
        "display_width_px": "1024",
        "display_number": "1",
    },
)

# Step 3.2: Create and connect bash motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="BashActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Bash",
    parentActionGroupSignatureParams={
        "kind": "bash_20250124",
    },
)

# Step 3.3: Create and connect textual content editor motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="TextEditorActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.TextEditor",
    parentActionGroupSignatureParams={
        "kind": "text_editor_20250124",
    },
)

# Step 3.4 Create Climate Motion Group

bedrock_agent.create_agent_action_group(
        actionGroupName="WeatherActionGroup",
        agentId=create_agent_response["agent"]["agentId"],
        agentVersion="DRAFT",
        actionGroupExecutor = {
            'customControl': 'RETURN_CONTROL',
        },
        functionSchema = {
            'features': [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location.",
                    "parameters": {
                        "location": {
                            "type": "string",
                            "description": "The city, e.g., San Francisco",
                            "required": True,
                        },
                        "unit": {
                            "type": "string",
                            "description": 'The unit to use, e.g., 
									fahrenheit or celsius. Defaults to "fahrenheit"',
                            "required": False,
                        },
                    },
                    "requireConfirmation": "DISABLED",
                }
            ]
        },
)
time.sleep(10)
# Step 4: Put together agent

bedrock_agent.prepare_agent(agentId=create_agent_response["agent"]["agentId"])

Instance use case

On this put up, we display an instance the place we use Amazon Bedrock Brokers with the pc use functionality to finish an internet type. Within the instance, the pc use agent may change Firefox tabs to work together with a buyer relationship administration (CRM) agent to get the required data to finish the shape. Though this instance makes use of a pattern CRM software because the system of file, the identical strategy works with Salesforce, SAP, Workday, or different methods of file with the suitable authentication frameworks in place.

Within the demonstrated use case, you possibly can observe how effectively the Amazon Bedrock agent carried out with laptop use instruments. Our implementation accomplished the client ID, buyer title, and electronic mail by visually analyzing the excel information. Nevertheless, for the overview, it determined to pick out the cell and replica the information, as a result of the data wasn’t utterly seen on the display screen. Lastly, the CRM agent was used to get further data on the client.

Greatest practices

The next are some methods you possibly can enhance the efficiency in your use case:

Issues

The pc use function is made obtainable to you as a beta service as outlined within the AWS Service Phrases. It’s topic to your settlement with AWS and the AWS Service Phrases, and the relevant mannequin EULA. Pc use poses distinctive dangers which can be distinct from customary API options or chat interfaces. These dangers are heightened when utilizing the pc use function to work together with the web. To reduce dangers, contemplate taking precautions reminiscent of:

• Function laptop use performance in a devoted digital machine or container with minimal privileges to reduce direct system exploits or accidents
• To assist stop data theft, keep away from giving the pc use API entry to delicate accounts or information
• Restrict the pc use API’s web entry to required domains to scale back publicity to malicious content material
• To implement correct oversight, preserve a human within the loop for delicate duties (reminiscent of making selections that might have significant real-world penalties) and for something requiring affirmative consent (reminiscent of accepting cookies, executing monetary transactions, or agreeing to phrases of service)

Any content material that you just allow Anthropic’s Claude to see or entry can probably override directions or trigger the mannequin to make errors or carry out unintended actions. Taking correct precautions, reminiscent of isolating Anthropic’s Claude from delicate surfaces, is crucial – together with to keep away from dangers associated to immediate injection. Earlier than enabling or requesting permissions essential to allow laptop use options in your personal merchandise, inform finish customers of any related dangers, and procure their consent as applicable.

Clear up

If you end up performed utilizing this answer, make certain to wash up all of the assets. Comply with the directions within the offered GitHub repository.

Conclusion

Organizations throughout industries face important challenges with cross-application workflows that historically require guide information entry or advanced customized integrations. The mixing of Anthropic’s laptop use functionality with Amazon Bedrock Brokers represents a transformative strategy to those challenges.

Through the use of Amazon Bedrock Brokers because the orchestration layer, organizations can alleviate the necessity for customized API improvement for every software, profit from complete logging and tracing capabilities important for enterprise deployment, and implement automation options shortly.

As you start exploring laptop use with Amazon Bedrock Brokers, contemplate workflows in your group that might profit from this strategy. From bill processing to buyer onboarding, HR documentation to compliance reporting, the potential purposes are huge and transformative.

We’re excited to see how you’ll use Amazon Bedrock Brokers with the pc use functionality to securely streamline operations and reimagine enterprise processes via AI-driven automation.

Assets

To be taught extra, confer with the next assets:

Concerning the Authors

Eashan Kaushik is a Specialist Options Architect AI/ML at Amazon Net Providers. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric strategy to his work. Earlier than this function, he obtained an MS in Pc Science from NYU Tandon Faculty of Engineering. Outdoors of labor, he enjoys sports activities, lifting, and working marathons.

Maira Ladeira Tanke is a Tech Lead for Agentic workloads in Amazon Bedrock at AWS, the place she allows prospects on their journey to develop autonomous AI methods. With over 10 years of expertise in AI/ML. At AWS, Maira companions with enterprise prospects to speed up the adoption of agentic purposes utilizing Amazon Bedrock, serving to organizations harness the ability of basis fashions to drive innovation and enterprise transformation. In her free time, Maira enjoys touring, taking part in together with her cat, and spending time together with her household someplace heat.

Raj Pathak is a Principal Options Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance coverage, Capital Markets) prospects throughout Canada and the US. Raj makes a speciality of Machine Studying with purposes in Generative AI, Pure Language Processing, Clever Doc Processing, and MLOps.

Adarsh Srikanth is a Software program Growth Engineer at Amazon Bedrock, the place he develops AI agent providers. He holds a grasp’s diploma in laptop science from USC and brings three years of trade expertise to his function. He spends his free time exploring nationwide parks, discovering new mountain climbing trails, and taking part in varied racquet sports activities.

Abishek Kumar is a Senior Software program Engineer at Amazon, bringing over 6 years of priceless expertise throughout each retail and AWS organizations. He has demonstrated experience in creating generative AI and machine studying options, particularly contributing to key AWS providers together with SageMaker Autopilot, SageMaker Canvas, and AWS Bedrock Brokers. All through his profession, Abishek has proven ardour for fixing advanced issues and architecting large-scale methods that serve hundreds of thousands of shoppers worldwide. When not immersed in expertise, he enjoys exploring nature via mountain climbing and touring adventures together with his spouse.

Krishna Gourishetti is a Senior Software program Engineer for the Bedrock Brokers group in AWS. He’s enthusiastic about constructing scalable software program options that resolve buyer issues. In his free time, Krishna likes to go on hikes.