Introduction
Imagine building an AI assistant for an enterprise application.
The assistant can read customer information, update records, generate reports, and even perform administrative operations through internal APIs.
Everything works perfectly during testing.
A user asks the AI assistant to update a customer's information, and the agent identifies the correct tool, generates the required arguments, and sends the request to the backend.
But what happens when the same user asks the AI to delete a customer record they are not authorized to modify?
The AI understands the instruction. It knows which tool to call.
But should it be allowed to execute that operation?
This is where building an AI agent becomes more than just an LLM integration problem. It becomes a software security and architecture problem.
As AI agents become more capable, developers need to think carefully about what these systems can access and what actions they are allowed to perform.
An AI agent's ability to execute an action should never be confused with its authority to execute that action.
1. From Chatbots to AI Agents
Traditional chatbots were mainly designed to answer questions and provide information based on predefined responses or language models.
Modern AI agents can do much more.
They can interact with external tools, call APIs, retrieve information from databases, execute functions, and perform multi-step operations.
For example, imagine asking an AI assistant:
"Find customer details for customer ID 1024 and update their contact information."
The agent might follow this process:
- Understand the user's request.
- Identify the required customer information.
- Select the appropriate tool.
- Generate the required arguments.
- Call the backend API.
- Return the result to the user.
This is commonly known as Tool Calling.
Instead of simply generating a text response, the model can request that an application execute a particular function.
However, there is an important distinction.
The AI model decides which tool might be useful. The application is responsible for validating and executing that request.
That distinction is extremely important when developing enterprise AI systems.
2. The Hidden Security Problem
Let's consider a simple example.
An enterprise application has three types of users:
- Administrator: Can view, update, and delete customer records.
- Manager: Can view and update customer records.
- Employee: Can only view customer records.
Now imagine that an AI agent has access to three tools:
tools = [
"view_customer",
"update_customer",
"delete_customer"
]
If every user can access the AI assistant, but the backend does not enforce individual permissions, the agent might execute operations that the user should never be allowed to perform.
This creates several security risks.
Unauthorized actions: An agent might execute operations beyond the user's permissions.
Prompt injection: Malicious instructions hidden in user input or retrieved documents might influence the agent's tool selection.
Excessive permissions: The agent might have access to more functionality than its actual task requires.
Sensitive information exposure: An agent might retrieve information belonging to another user or organization.
This is not just a theoretical concern.
OWASP identifies this category of risk as Excessive Agency in its 2025 Top 10 for LLM Applications.
The underlying problem is that AI systems may receive excessive functionality, excessive permissions, or excessive autonomy.
The solution is not simply to write better system prompts. Security controls must exist outside the language model.
3. Understanding the Principle of Least Privilege
One of the most important principles in application security is the Principle of Least Privilege.
In simple terms:
Give a system only the permissions it needs to perform its intended task.
The same principle should apply to AI agents.
For example, if an AI assistant is designed to generate customer reports, does it really need permission to delete customer records?
Probably not.
Instead of providing access to every available operation, expose only the tools required for the agent's responsibilities.
Consider this approach:
| AI operation | Required permission |
|---|
| View customer | customer |
| Update customer | customer |
| Delete customer | customer |
| Generate report | report |
This makes permissions easier to understand and manage.
It also reduces the potential damage if an agent makes an incorrect decision or receives manipulated instructions.
Another important consideration is user-level access.
If an employee can only access records belonging to their assigned department, the AI assistant must follow the same restriction.
The agent should not receive broader access simply because it is an AI system.
4. Backend Authorization: The Most Important Layer
This is where backend engineering becomes extremely important.
A common mistake is to depend on the AI model to decide whether a user is authorized to perform an operation.
For example, developers might add instructions to a system prompt:
"Never delete customer records unless the user is an administrator."
This instruction is useful, but it is not a security mechanism.
The model can misunderstand instructions, produce unexpected outputs, or be influenced by prompt injection.
Authorization must be enforced by the backend.
Let's understand this with a simple Python example.
Step 1: Define the permissions
ROLE_PERMISSIONS = {
"admin": {
"customer:read",
"customer:write",
"customer:delete"
},
"manager": {
"customer:read",
"customer:write"
},
"employee": {
"customer:read"
}
}
Each role has a defined set of permissions.
Step 2: Create an authorization function
def check_permission(user, required_permission):
permissions = ROLE_PERMISSIONS.get(
user.role, set()
)
if required_permission not in permissions:
raise PermissionError(
"You are not authorized to perform this action."
)
return True
This function checks whether the authenticated user has the required permission.
Step 3: Validate the tool execution
def execute_tool(user, tool_name, arguments):
tool_permissions = {
"view_customer": "customer:read",
"update_customer": "customer:write",
"delete_customer": "customer:delete"
}
required_permission = tool_permissions.get(
tool_name
)
if not required_permission:
raise PermissionError("Unknown tool")
check_permission(user, required_permission)
return tool_registry[tool_name](
user=user,
**arguments
)
Now the backend validates the user's permissions before executing the requested operation.
Notice something important:
The AI agent can suggest delete_customer, but it cannot grant itself permission to execute that operation.
The backend remains responsible for the final decision.
Note: These snippets demonstrate the authorization concept. A production implementation should also use trusted authentication, strict input schemas, resource-level authorization, tenant isolation, database transactions, and secure error handling.
5. Human Approval and Audit Logging
Not every operation should execute immediately, even when the user has permission.
Consider these operations:
- Deleting a customer account.
- Changing financial information.
- Triggering a production deployment.
- Modifying critical system configurations.
These actions can have significant consequences.
A better approach is to introduce a human approval mechanism for sensitive operations.
For example:
User: Delete customer record 1024.
AI Agent: I have identified the customer record. This operation requires administrator approval.
Backend: Creates an approval request and waits for confirmation.
Administrator: Approves the operation.
Backend: Revalidates the approval and authorization, then executes the operation.
This creates an additional layer of control.
However, approval should not replace authorization. The backend must still verify that the approving person is authorized, that the approval is valid, and that the operation being executed matches the approved request.
Why Audit Logging Matters
Every important tool execution should be traceable.
A useful audit record might contain:
{
"actor_id": 104,
"tool_name": "update_customer",
"resource_id": 1024,
"action": "CUSTOMER_UPDATE",
"status": "SUCCESS",
"approval_required": false
}
In a production system, you should also consider recording:
- Request or correlation ID.
- Timestamp.
- Tenant or organization ID.
- Authorization decision.
- Approval reference, where applicable.
- Execution result.
Avoid storing passwords, access tokens, or unnecessary sensitive prompt content in audit logs.
Audit logging helps developers investigate unexpected operations and understand what happened during an AI agent's execution.
6. Designing Secure Enterprise AI Agents
Building a secure AI agent requires more than connecting an LLM to a collection of APIs.
The application needs clearly defined trust boundaries.
A practical architecture looks like this:
User → AI Agent → Tool Execution Gateway → Backend Authorization → Business Logic → Database
Each component has a different responsibility.
AI Agent: Understands the user's request and selects an appropriate tool.
Tool Execution Gateway: Validates tool names, arguments, and execution requests.
Backend Authorization: Checks the authenticated user's permissions and resource-level access.
Business Logic: Executes the approved operation according to application rules.
Audit Logging: Records important actions and their outcomes.
There are several additional practices worth considering:
- Use separate permissions for read and write operations.
- Avoid unrestricted tools such as arbitrary SQL execution or shell commands.
- Validate every tool argument before execution.
- Enforce authorization at the actual backend service, not only at the AI gateway.
- Require approval for sensitive or irreversible operations.
- Apply rate limits and execution limits to prevent misuse.
- Test direct and indirect prompt-injection scenarios.
Security should be designed into the application architecture rather than added after the AI agent has already been deployed.
Conclusion
AI agents are changing how applications interact with users and external systems.
They can retrieve information, execute operations, and automate workflows that previously required multiple manual steps.
But increased capability also introduces increased responsibility.
As developers, we should not assume that an AI model will always make the correct decision or follow every security instruction.
A well-designed AI system should separate decision-making from authorization.
The model can suggest an action, but the backend must validate whether that action is permitted.
Ultimately, building intelligent AI applications is not just about making agents more capable.
It's about making them capable without giving them unnecessary authority.
What do you think? Should AI agents operate with the same permissions as the user, or should they have a separate permission system designed specifically for AI-driven operations?
Join the conversation! Your thoughts help the community grow.