Developers spend a lot of time working outside their code editor. They may need to update information in a desktop application, copy data between tools, review reports, or complete workflows in software that does not provide an API.
Traditionally, GitHub Copilot has focused on helping developers write and understand code. Agent mode extended this capability by allowing Copilot to edit project files, run terminal commands, and work through multi-step development tasks.
Computer Use takes this further. It allows GitHub Copilot to interact with desktop applications through their graphical interfaces. It can read visible application content, click buttons, enter text, press keys, scroll, and move through workflows.
This capability is useful when an application has no API, command-line interface, or MCP integration. Instead of manually repeating the same steps, developers can describe the desired result and allow Copilot to work through the interface with permission.
Computer Use became available in public preview in October 2026 for GitHub Copilot CLI and the GitHub Copilot app on macOS and Windows. It is also available experimentally in local GitHub Copilot sessions in Visual Studio Code.
Let's understand how it works and how developers can use it safely.
What Is Computer Use in GitHub Copilot?
Computer Use is an agent capability that allows Copilot to interact with applications running on a user's computer.
It uses operating-system accessibility information and screenshots when visual context is needed to understand an application window.
Once enabled, Copilot can perform actions such as:
Read accessible text and application content.
Identify buttons, menus, and other interface controls.
Click controls and enter text.
Edit information in forms and documents.
Press keyboard shortcuts.
Scroll through application windows.
Drag items and navigate between application screens.
Complete workflows involving multiple desktop applications.
For example, consider a developer who needs to review an expense report in a desktop application, update a presentation, and copy selected information into another business tool.
If these applications do not expose suitable APIs or automation interfaces, Copilot can interact with them through their graphical interfaces.
The important difference is that Copilot is no longer limited to the editor or terminal. It can work with other applications that the user explicitly allows it to access.
How Computer Use Works
Computer Use adds a visual interaction layer to the existing agent workflow.
A typical interaction follows this process:
The developer describes the task.
Copilot identifies the applications involved.
Copilot requests permission to control the required application.
The user approves or rejects the request.
Copilot reads the available interface information.
The agent selects controls and performs the requested actions.
The user reviews the result.
For example, suppose a developer wants to update a project status in a desktop project management application.
The developer can provide this instruction:
Open the project management application.
Find the project named Customer Portal Upgrade.
Change its status to In Progress.
Do not modify the project owner, deadline,
or any other field.
Before saving, show me the changes.
Copilot can navigate the application, locate the project, and update the requested field.
The instruction includes an important constraint: only the project status should change.
Clear boundaries help reduce accidental modifications when an agent works with an interface containing many editable fields.
Where Computer Use Is Available
Computer Use is available across several GitHub Copilot surfaces, but the setup and feature status differ.
Copilot surface | Availability | Operating systems |
|---|
GitHub Copilot CLI | Public preview | macOS and Windows |
GitHub Copilot app | Public preview | macOS and Windows |
GitHub Copilot in VS Code | Experimental | macOS and Windows |
In VS Code, Computer Use requires a local Copilot session. It is not available for remote or cloud hosts, Linux, or other agent harnesses that do not support this capability.
The feature is disabled by default. Organization administrators can also disable it through managed settings.
Enable Computer Use in GitHub Copilot CLI
GitHub Copilot CLI provides a terminal-based interface for working with an AI agent.
It can already inspect files, edit code, and execute commands. Computer Use adds the ability to interact with desktop applications.
Step 1: Start GitHub Copilot CLI
Open a terminal and start a Copilot CLI session:
copilot
Sign in with your GitHub account if required.
If Copilot CLI is provided through an organization, confirm that the organization administrator has enabled the CLI policy.
Step 2: Enable Computer Use
Inside the Copilot CLI session, execute:
/computer on
Copilot enables the Computer Use capability and guides you through any required setup.
You can check its current status with:
/computer show
To disable it later, run:
/computer off
These commands control whether the agent can use the desktop interaction capability.
Step 3: Approve application access
When Copilot attempts to control an application, it follows the configured tool permission settings.
You may be asked to approve access for the current session or allow the application for future sessions.
Choose the current-session option when testing an application for the first time.
Avoid granting permanent access to applications that contain sensitive financial, customer, or enterprise information unless there is a clear operational reason.
Enable Computer Use in the GitHub Copilot App
The GitHub Copilot app provides a graphical interface for coordinating agent tasks.
To enable Computer Use:
Open the GitHub Copilot app.
Open Settings.
Select Computer Use from the sidebar.
Find Enable Computer Use.
Turn on the feature.
Complete the required operating-system permissions.
On macOS, Copilot requires two permissions:
You can also enable the feature from a session using:
/computer on
The application permission settings determine whether Copilot must request approval before controlling a desktop application.
If you choose Always allow for an application, the approval is saved locally and also applies to the Copilot CLI on the same computer.
Review these saved approvals periodically. Removing an application from the always-allowed list prevents future automatic approvals, but it does not revoke access already granted in an active session.
Use Computer Use in Visual Studio Code
GitHub Copilot in VS Code also supports experimental Computer Use in local sessions.
This is useful when a developer wants to work with code and desktop applications without switching to a separate Copilot surface.
To enable it:
Open Visual Studio Code.
Open Settings.
Search for chat.agentHost.computerUse.enabled.
Enable the setting.
Restart VS Code.
Start a new local Copilot chat.
Run /computer on in the Copilot session.
The first time the capability is used, the required plugin is downloaded automatically.
When Copilot requests access to an application, review the application name and requested action before approving it.
This feature is experimental, so availability and behavior may change as GitHub and VS Code continue development.
Practical Example: Updating a Desktop Application
Consider a development team that maintains a customer support application. Every morning, a developer reviews an incident report in a desktop application and updates a project tracking tool.
The workflow normally involves:
Opening the incident report.
Finding unresolved issues.
Reading the assigned team and priority.
Opening the project tracking application.
Finding the corresponding task.
Updating its status.
Adding a short comment.
If these applications do not provide suitable integration points, Computer Use can help automate the workflow.
A carefully written instruction could look like this:
Open the incident management application.
Find unresolved incidents assigned to
the Backend team.
For each incident, read its title, priority,
and current status.
Open the project tracking application and
find the matching task.
If a matching task exists, add the incident
priority to its comment.
Do not close incidents or change task status.
Show me the changes before saving.
This instruction defines the applications, selection criteria, required action, and restrictions.
The developer should still verify that Copilot selected the correct incident and matching task. Similar titles or outdated records can cause incorrect matches.
For a workflow that changes production records, keep approval enabled and review each modification before submission.
Write Better Instructions for Desktop Automation
Computer Use depends heavily on the clarity of the task description.
A vague instruction such as "Update my project" does not identify which project, which field, or what value should be used.
A useful prompt should contain four elements.
Identify the application
Tell Copilot which application it should open.
For example:
Open the desktop application named Inventory Manager.
Define the target
Specify the record, window, document, or item that should be modified.
Find the product with SKU LAP-1045.
Describe the expected action
Explain exactly what should change.
Update its available quantity to 25.
Add restrictions
State what Copilot must not modify.
Do not change the product price, description,
or supplier information.
These details reduce ambiguity and make it easier to review the result.
For important workflows, ask Copilot to inspect the current state first and report the proposed changes before saving anything.
Security and Permission Management
Computer Use can interact with applications that contain personal and business information. This makes permission management an important part of its implementation.
The capability is disabled by default, and users must enable it before an agent can access desktop applications.
However, enabling the feature does not mean every application should receive unrestricted access.
Follow these practices:
Grant access only to applications required for the task.
Prefer session-based approval when testing new workflows.
Avoid permanent approval for banking, payroll, and other sensitive applications.
Review the requested action before allowing it.
Avoid displaying passwords, authentication codes, or confidential customer records.
Stop the agent if it begins interacting with an unexpected window.
Review modified records before submitting important changes.
In the GitHub Copilot app, use Stop or press Esc to interrupt an active operation.
In Copilot CLI, press Esc twice to interrupt the operation.
Organization-managed policies take precedence over local approval settings. Users cannot enable Computer Use locally if an organization has disabled the capability.
Limitations Developers Should Understand
Computer Use is useful, but graphical interfaces introduce problems that do not usually occur with structured APIs.
Interface changes can break workflows
A button may move after an application update. A menu may have a different layout, or a dialog may appear unexpectedly.
The agent may fail to identify the correct control or may need additional instructions.
Similar controls can cause mistakes
An application might contain multiple buttons with the same label. The agent may select the wrong one if the surrounding context is unclear.
This is especially important when the action deletes records, submits transactions, or changes production information.
Dynamic applications can be difficult to automate
Some applications load information asynchronously, display custom controls, or require several authentication steps.
These behaviors can make interaction less predictable.
Computer Use is not always the best tool
If an application provides a stable API, MCP server, command-line interface, or dedicated browser integration, use that interface when possible.
Structured integrations generally provide more predictable inputs and outputs than visual interaction.
Computer Use is most useful when there is no suitable direct integration.
Computer Use vs. Traditional Copilot Agent Mode
Computer Use and agent mode solve different problems.
Capability | Traditional agent mode | Computer Use |
|---|
Edit project files | Yes | Not its primary purpose |
Run terminal commands | Yes | No |
Read desktop application content | Not directly | Yes |
Click application controls | Not directly | Yes |
Enter text into desktop forms | Not directly | Yes |
Work across graphical applications | Limited without integrations | Yes |
Access application APIs and MCP tools | Yes, when configured | Not required for visual interaction |
Traditional agent mode is suitable for development tasks involving source code, tests, and terminal commands.
Computer Use is useful when a task requires interaction with graphical applications that the agent cannot access through other tools.
They can complement each other in a broader workflow, but developers should choose the most reliable tool for each operation.
Summary
GitHub Copilot Computer Use extends AI-assisted development beyond code editors and terminals. It allows agents to read and interact with desktop applications by clicking controls, entering text, navigating screens, and completing multi-step workflows.
Developers can use it through GitHub Copilot CLI and the GitHub Copilot app on macOS and Windows. An experimental implementation is also available in local GitHub Copilot sessions in VS Code.
The feature is particularly useful for legacy software and applications without APIs or command-line support. However, graphical automation is less predictable than structured integrations.
Start with simple, low-risk tasks, grant access only to the required applications, and review every important change before saving it. Computer Use can reduce repetitive work, but developers remain responsible for the actions performed on their systems.
Join the conversation! Your thoughts help the community grow.