Most developers are already comfortable asking GitHub Copilot to write code, explain an error, create tests, or work with files. But there is a different kind of task that code-focused AI assistants have traditionally struggled with: operating a desktop application.

For example, imagine that you have an old Windows application with no API, no command-line interface, and no MCP server. You need to open the application, find a particular record, copy information, update a field, and move the result into another application.

Normally, a developer has to do all of those clicks manually.

GitHub Copilot Computer Use is designed for this type of problem. It allows Copilot to interact with desktop applications by reading visible application content, clicking controls, entering text, pressing keys, scrolling, dragging items, and moving through multi-step workflows.

The feature is currently available in public preview through GitHub Copilot CLI and the GitHub Copilot app on supported macOS and Windows environments. It is important to understand what this capability can and cannot do before using it for real work.

What Is GitHub Copilot Computer Use?

GitHub Copilot Computer Use gives an AI agent the ability to interact with applications through their user interface.

Traditional AI coding assistance usually looks like this:

Developer
   |
   v
Copilot
   |
   +-- Read code
   +-- Generate code
   +-- Modify files
   +-- Run commands

Computer Use adds another layer:

Developer
   |
   v
Copilot Agent
   |
   v
Desktop Application
   |
   +-- Read application content
   +-- Click controls
   +-- Enter text
   +-- Press keys
   +-- Scroll
   +-- Drag items
   +-- Navigate workflows

This is useful when the application does not expose a clean programmatic interface.

For example, a company may still have an internal desktop application built years ago. If that application has no REST API or CLI, writing a traditional integration can be expensive or simply impossible.

Computer Use gives an agent another way to interact with it.

What Can Copilot Actually Do?

Computer Use is not just about reading the screen. Copilot can perform several types of user-interface actions.

Read Information From an Application

Copilot can inspect accessible application content and, when necessary, use visual context from the application window.

For example, you could ask:

Open the expense application and summarize the pending
expense reports shown on the main screen.

Do not modify anything or submit any forms.

This is a relatively safe starting point because the task is read-only.

Click Buttons and Controls

Copilot can interact with buttons, menus, checkboxes, and other controls.

For example:

Open the desktop inventory application.

Navigate to the Products section and find products
that are marked as inactive.

Do not change any product information.

The important part is that the instruction describes the outcome and the application involved.

Enter and Edit Text

Computer Use can enter information into fields and edit existing text.

A simple workflow might look like:

Open the customer management application.

Find customer ID 1042.
Open the customer profile.
Update the phone number to the value provided in this task.
Stop before saving the record.

The final instruction is useful because it creates a human approval point before a potentially important change.

Navigate Multi-Step Workflows

This is where the feature becomes more interesting.

Some desktop applications require several screens to complete a simple operation:

  1. Open the application.

  2. Select a module.

  3. Search for a record.

  4. Open the record.

  5. Change a value.

  6. Move to another screen.

  7. Submit the operation.

Computer Use can navigate these workflows instead of requiring the developer to describe every individual click.

Where Computer Use Makes the Most Sense

The strongest use case is not replacing APIs. It is handling applications where APIs or other structured interfaces are unavailable.

Consider an old internal application:

Legacy Desktop Application
        |
        +-- No REST API
        +-- No CLI
        +-- No MCP server
        +-- GUI only

A traditional integration may require reverse engineering or building additional infrastructure.

Computer Use provides another option:

Copilot
   |
   v
Desktop GUI
   |
   +-- Search
   +-- Read
   +-- Update
   +-- Navigate

Typical use cases include legacy business applications, GUI-only tools, desktop reporting workflows, presentation editing, and moving information between applications.

However, if the same task can be completed through an API, command-line tool, filesystem operation, or MCP server, the structured approach is generally better.

Computer Use vs Traditional Automation

It is useful to compare Computer Use with the approaches developers already use.

Approach

Best For

Main Strength

Main Limitation

REST API

Application integration

Structured and predictable

API must exist

CLI

Developer automation

Easy to script

CLI must exist

MCP

AI tool integration

Structured AI access to tools

MCP server must exist

UI automation

GUI workflows

Works with visual applications

Can break when UI changes

Copilot Computer Use

AI-driven GUI workflows

Natural-language task execution

Visual interfaces can be unpredictable

This distinction matters in production.

If an application provides a reliable API, using the API is normally preferable to asking an AI agent to click through the application.

Computer Use is most valuable when the GUI is the only practical interface.

How to Enable Computer Use

Computer Use is disabled by default.

In the GitHub Copilot app, the feature can be enabled from the Computer Use settings. Copilot CLI also provides a command-based option.

For CLI sessions, the computer-use capability can be enabled with:

/computer on

You can check its status with:

/computer show

And disable it with:

/computer off

On macOS, additional operating-system permissions are required so Copilot can interact with applications and inspect application windows when visual context is needed.

Before enabling the feature, developers should understand what applications they are allowing Copilot to control.

Writing Better Computer Use Prompts

The quality of the instruction matters.

A weak instruction might be:

Update the application.

There is no clear target, application, expected result, or safety boundary.

A better instruction is:

Open the inventory application.

Find product ID 1042 and open its details.

Change the reorder level from 10 to 20.

Do not change any other fields.

Stop before saving the record so I can review the change.

This prompt gives Copilot:

For production workflows, these details are important.

A Safe Example Workflow

Suppose a support team uses a desktop application to review customer tickets.

A developer could start with a read-only task:

Open the support application.

Find the list of unresolved tickets.

Read the ticket titles and priority values.

Give me a summary of the five highest-priority tickets.

Do not edit, close, or submit any ticket.

Once the workflow is understood, a controlled modification could be tested:

Open the support application.

Find ticket 5821.

Open the ticket and change its priority to High.

Do not close the ticket.

Stop before submitting the change.

This approach is safer than immediately giving an agent unrestricted access to an application.

Security Considerations

Computer Use introduces a different security model from normal code generation.

The agent can interact with applications that may contain sensitive information. That could include customer records, financial information, internal systems, or business data.

A developer should therefore treat desktop access as a privileged capability.

One important rule is to avoid giving unrestricted access unless it is genuinely required.

For example, prefer:

Read the report and summarize the totals.
Do not modify the report.

over:

Open the application and handle everything required.

The first instruction establishes a clear boundary.

Copilot also provides approval controls for application access. Developers can review requested actions and stop an active operation when necessary.

For sensitive applications, avoid automatically allowing access for future sessions unless the organization has reviewed the risk.

Common Mistakes

Giving Too Much Permission

The first mistake is enabling Computer Use and immediately allowing access to every application.

Start with one low-risk application and a limited workflow.

Using It When an API Exists

If an application exposes a stable API, use it.

For example, retrieving customer information through an API is generally more predictable than asking an agent to open a customer-management application and navigate through several screens.

Giving Ambiguous Instructions

Avoid prompts such as:

Fix the records.

Instead specify:

Find records with status "Pending Review".

Show me the matching records first.

Do not make any changes.

Ignoring UI Changes

Computer Use relies on the application's interface.

If a button moves, a menu changes, a dialog appears, or an application behaves differently after an update, the workflow may no longer behave as expected.

This is one of the biggest differences between GUI automation and API-based automation.

Troubleshooting Computer Use

When a workflow does not work as expected, check the following.

The Computer Use Option Is Missing

First verify that the feature is available for the current Copilot setup and operating system.

Because Computer Use is a preview capability, availability and behavior can change.

Copilot Cannot Control the Application

On macOS, check that the required Accessibility and Screen Recording permissions have been granted.

Also verify that Computer Use is enabled.

Copilot Clicks the Wrong Control

Make the prompt more specific.

Instead of:

Click the button.

use:

In the customer details window, click the Save Changes
button after reviewing the updated phone number.

If the application has several similar buttons, describe the surrounding context.

The Workflow Stops Halfway

Break a large task into smaller stages.

For example:

First, open the application and locate the report.

Do not edit anything.

Tell me when the report is open.

Then continue with the next operation.

This also makes debugging easier.

Advantages of GitHub Copilot Computer Use

Works With GUI-Only Applications

The biggest advantage is the ability to work with applications that lack APIs, CLIs, or MCP integrations.

Useful for Legacy Systems

Organizations often have older applications that are still important but difficult to integrate with modern systems.

Computer Use can provide another automation option.

Natural-Language Workflows

Developers can describe the desired outcome instead of manually scripting every click.

Useful Across Applications

The agent can navigate workflows that involve multiple desktop applications rather than being limited to one coding environment.

Disadvantages and Limitations

UI Changes Can Break Workflows

A visual interface is less stable than a well-defined API.

Actions Can Be Unpredictable

An agent may misunderstand a visual layout, select the wrong control, or encounter an unexpected dialog.

Security Risk

The ability to control applications means the agent can potentially affect real data and connected accounts.

Human Review Is Still Important

Computer Use should not be treated as a replacement for human judgment, especially when an action can modify important data or affect other people.

Best Practices for Production Use

If you plan to experiment with Computer Use in a real development environment, follow a few simple rules.

  1. Start with read-only workflows.
    Begin with tasks that collect or summarize information.

  2. Use the smallest required permissions.
    Give access only to the applications needed for the task.

  3. Prefer APIs and structured tools when available.
    Use Computer Use mainly when direct integrations are unavailable or impractical.

  4. Add approval points before important changes.
    Stop before submitting payments, deleting records, changing permissions, or performing other high-impact actions.

  5. Write explicit prompts.
    Define the application, target, expected result, and restrictions.

  6. Test against UI changes.
    A workflow that works today may need adjustment after an application update.

  7. Avoid sensitive applications during experimentation.
    First test the workflow with non-sensitive data.

  8. Keep a human in the loop for high-impact actions.
    Automation should reduce repetitive work, not remove necessary oversight.

Final Thoughts

GitHub Copilot Computer Use changes the type of work an AI coding agent can perform. Instead of being limited to code, files, and terminal commands, Copilot can interact with desktop applications through their user interfaces.

That does not mean every desktop task should now be automated with AI.

For applications with reliable APIs, command-line interfaces, or structured integrations, those approaches remain easier to test and maintain. Computer Use becomes more interesting when the application is legacy, GUI-only, or otherwise difficult to integrate.

For developers, the practical opportunity is simple: identify the repetitive desktop workflows that currently require manual clicking and determine whether an AI agent can safely handle part of that process.

Start small, keep permissions limited, use read-only workflows first, and require human approval before important changes. That is a much more realistic way to evaluate Computer Use than treating it as a fully autonomous desktop operator.

Summary

GitHub Copilot Computer Use allows Copilot to interact with desktop applications by reading application content, clicking controls, entering text, pressing keys, scrolling, dragging, and navigating workflows.

Its main value is in GUI-only and legacy applications where APIs, CLI tools, or MCP integrations are not available. At the same time, visual automation is less predictable than structured integrations, so developers should use clear prompts, limited permissions, and human approval for sensitive actions.

The feature is best viewed as another tool in the developer's automation toolkit, not a replacement for APIs or normal software engineering practices.