Managing External API Secrets With AWS Secrets Manager

Reducing Repeated Secrets Manager API Calls

One important consideration is how frequently an application retrieves secrets.

Calling Secrets Manager every time an API request is processed is usually unnecessary.

For example, an application receiving thousands of requests may repeatedly retrieve the same API key.

This increases network calls, adds latency, and can increase Secrets Manager costs.

A better approach is to cache the retrieved secret for a controlled period.

Using a Simple Application-Level Cache

The following example keeps a secret in application memory for five minutes.

import time

from secrets_manager import get_secret

_cached_secret = None
_cache_expiry = 0

def get_cached_payment_secret():

    global _cached_secret, _cache_expiry

    current_time = time.time()

    if (
        _cached_secret is None
        or current_time >= _cache_expiry
    ):
        _cached_secret = get_secret(
            "production/payment-service",
            "ap-south-1"
        )

        _cache_expiry = current_time + 300

    return _cached_secret

The application retrieves the secret when the cache is empty or expired.

Subsequent calls reuse the cached value until the five-minute period ends.

This is a simple implementation. In applications with multiple threads, use appropriate synchronization. For larger Python applications, the AWS Secrets Manager caching component is another option.

AWS provides a Python caching library called aws-secretsmanager-caching.

Install it using:

pip install aws-secretsmanager-caching

The library supports client-side caching and refreshes cached secrets periodically. Its default refresh interval is one hour.

Caching introduces an important trade-off: an application may continue using an older credential until its cache refreshes. Choose a refresh interval based on the rotation schedule and the external service's credential requirements.

Rotating External API Credentials

Storing credentials securely is only part of secret management. Teams must also replace credentials periodically and respond quickly when a credential is compromised.

For example, suppose a payment provider requires the application to replace its API key every 90 days.

Without a centralized secret management process, developers might need to update multiple configuration files and redeploy several applications.

AWS Secrets Manager supports secret rotation through scheduled rotation and Lambda-based rotation functions.

However, rotating an external API key requires cooperation from the external provider. Secrets Manager cannot automatically generate a new valid credential inside an unrelated third-party service unless that service provides an appropriate API or rotation mechanism.

Understanding the Rotation Process

A typical rotation workflow looks like this:

Rotation Schedule
       |
       v
Secrets Manager
       |
       v
Rotation Lambda Function
       |
       v
External API Provider
       |
       v
Generate New Credential
       |
       v
Validate New Credential
       |
       v
Update Secret Version
       |
       v
Application Retrieves New Secret

The rotation Lambda function communicates with the external service, creates or retrieves a replacement credential, and validates it before completing the rotation.

For supported rotation workflows, Secrets Manager uses staging labels such as AWSPENDING and AWSCURRENT to manage secret versions.

Implementing External API Key Rotation

The exact implementation depends on the external provider.

For example, a provider might offer an API endpoint that creates a new API token.

The rotation function would generally perform these operations:

1. Authenticate with the provider using the existing credential or another approved administrative identity.
2. Request a new API credential.
3. Store the new value as a pending secret version.
4. Test the new credential against the provider.
5. Promote the new version to the current version.
6. Revoke the old credential when it is safe to do so.

The order is important.

If the old credential is revoked before the new credential is validated, the application may lose access to the external service.

Some providers allow two active API keys during a transition. This makes it easier to update applications without interrupting requests.

If the provider supports only one active key, plan the rotation carefully and account for application cache expiration.

Handling Credential Rotation in the Application

An application should not assume that its cached credential remains valid indefinitely.

After rotation, the application should retrieve the current secret version when its cache expires.

For applications that need faster recovery, consider handling authentication failures by refreshing the secret and retrying a request when the provider's error response clearly indicates an expired or revoked credential.

Do not retry every failed request automatically. Network failures, invalid payloads, rate limits, and authorization errors require different handling.

Handling Secret Retrieval Errors in Production

Applications should be prepared for situations where Secrets Manager cannot return a credential.

Possible causes include:

• Missing IAM permissions.
• An incorrect secret name.
• An invalid AWS Region.
• Network connectivity problems.
• KMS decryption permission failures.
• A deleted or unavailable secret.

A production application should handle these situations without exposing secret values or internal AWS error details to users.

For example, improve the retrieval function with explicit error handling:

import boto3
import json

from botocore.exceptions import ClientError

def get_secret(secret_name, region_name):

    client = boto3.client(
        "secretsmanager",
        region_name=region_name
    )

    try:
        response = client.get_secret_value(
            SecretId=secret_name
        )

        value = response.get("SecretString")

        if value is None:
            raise RuntimeError(
                "Secret value is not available as text"
            )

        return json.loads(value)

    except client.exceptions.ResourceNotFoundException:
        raise RuntimeError(
            "Required application secret was not found"
        )

    except client.exceptions.AccessDeniedException:
        raise RuntimeError(
            "Application is not authorized to retrieve the secret"
        )

    except ClientError as error:
        raise RuntimeError(
            "AWS Secrets Manager request failed"
        ) from error

This example handles common retrieval failures separately.

In a real application, log the error category and relevant request information without logging the actual credential.

For temporary service failures, use bounded retries with exponential backoff and jitter where appropriate. Avoid retrying authorization failures or missing-secret errors indefinitely.

Monitoring and Auditing Secret Access

Knowing where credentials are stored is not enough. Teams should also understand which identities are retrieving them.

AWS provides CloudTrail integration for auditing Secrets Manager API activity.

Administrators can review events such as:

• Secret retrieval requests.
• Secret creation.
• Secret updates.
• Secret deletion.
• Rotation operations.
• Changes to secret resource policies.

For example, a security team may investigate why an application role retrieved a payment credential from an unexpected environment.

CloudTrail records help identify the AWS identity and API operation involved.

CloudTrail auditing does not record the secret value itself in the event details, which helps prevent credential exposure through the audit trail.

For production systems, configure appropriate monitoring and alerts for unusual access patterns, unexpected permission changes, and failed rotation operations.

Best Practices for Managing External API Secrets

Use Separate Secrets for Different Environments

Development and production applications should not share the same external API credentials.

Use separate secret names and IAM permissions for each environment.

Follow the Principle of Least Privilege

Give an application permission to retrieve only the secrets it requires.

Avoid granting broad access to every secret in an AWS account.

Prefer IAM Roles Over Static AWS Credentials

Use EC2 instance profiles, Lambda execution roles, or ECS task roles instead of storing long-term AWS access keys in application configuration.

Implement Client-Side Caching

Avoid retrieving the same credential for every application request.

Use an appropriate cache duration and make sure the application can obtain the updated credential after rotation.

Rotate Credentials Carefully

Use a rotation process supported by the external provider.

Validate new credentials before revoking old ones, and plan for the effect of application caching.

Avoid Logging Sensitive Information

Never log API keys, passwords, authorization headers, or complete secret values.

Review application logs, exception handlers, and monitoring integrations to ensure credentials are not accidentally exposed.

Monitor Access Through CloudTrail

Review secret access events and investigate unexpected retrieval activity.

This helps security teams understand how sensitive credentials are being used.

Plan for Secret Recovery

Document how to replace a compromised credential, restore access, and update dependent applications.

A secret management process should cover both routine rotation and emergency credential replacement.

Common Mistakes When Using AWS Secrets Manager

Advantages and Disadvantages of AWS Secrets Manager

Advantages

• Centralized storage for application credentials.
• Encryption at rest and secure retrieval.
• IAM-based access control.
• Integration with AWS services and SDKs.
• Support for secret versioning and rotation workflows.
• CloudTrail auditing of secret management operations.
• Client-side caching options for reducing repeated retrieval calls.

Disadvantages

• Secret retrieval and storage incur AWS charges.
• Applications require additional AWS permissions and configuration.
• External API rotation may require custom Lambda functions.
• Cached credentials can remain outdated until refreshed.
• Applications become dependent on Secrets Manager availability when retrieving credentials.
• Poorly configured IAM or KMS policies can prevent applications from accessing secrets.

Summary

Managing external API credentials is an important part of application security. Storing API keys directly in source code or sharing configuration files across environments creates unnecessary security and maintenance risks.

AWS Secrets Manager provides a centralized way to store, retrieve, encrypt, and manage these credentials. With Boto3, a Python application can retrieve its required API keys at runtime without placing the actual values inside the source code.

For production applications, secret retrieval should be combined with IAM least-privilege policies, client-side caching, credential rotation, and CloudTrail monitoring.

Remember that external API key rotation depends on the capabilities of the external provider. A complete rotation process must create and validate the replacement credential before the old one is revoked.

When these practices are implemented together, teams can manage external service credentials more consistently, reduce accidental exposure, and simplify application maintenance across development, staging, and production environments.