Apache Kafka applications exchange messages continuously between producers, consumers, and brokers. When these connections carry sensitive business information, encryption in transit is an important part of the system's security architecture.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) provides TLS encryption for communication between Kafka clients and brokers. However, upgrading an existing cluster to a newer TLS version requires more than changing a security setting. Different applications may use different Java versions, Kafka client libraries, operating systems, and TLS implementations.
An older application might support only TLS 1.2, while a newer application may already support TLS 1.3. Enforcing TLS 1.3 without checking client compatibility can interrupt message production and consumption.
Amazon MSK introduced TLS 1.3 support for MSK Provisioned clusters through configurable TLS security policies. The important part is that administrators can initially enable both TLS 1.2 and TLS 1.3, allowing applications to migrate without requiring every client to upgrade simultaneously.
This article explains how TLS 1.3 works with Amazon MSK, how to configure the compatibility policy, how to test Kafka clients, and how to migrate an existing production cluster safely.
Understanding TLS 1.3 in Amazon MSK
Transport Layer Security (TLS) protects communication between applications and servers by providing encryption, authentication, and message integrity.
Amazon MSK uses TLS to protect data transferred between Kafka clients and brokers. It also applies encryption policies to communication between the components managed by the MSK cluster.
TLS 1.3 improves the protocol by simplifying the handshake process, removing older cryptographic mechanisms, and supporting modern cipher suites.
Amazon MSK provides three security policies:
Security policy | TLS 1.2 | TLS 1.3 | Purpose |
|---|
MSK-TLS1.2-Default-2026 | Yes | No | Existing default configuration |
MSK-TLSv1.3-2026 | Yes | Yes | Compatibility during migration |
MSK-TLSv1.3-Only-2026 | No | Yes | Enforce TLS 1.3 exclusively |
For an existing production environment, MSK-TLSv1.3-2026 is generally the appropriate migration starting point.
It allows clients supporting only TLS 1.2 to continue connecting while newer clients negotiate TLS 1.3.
There is one important detail: a client that advertises TLS 1.3 but has a faulty TLS 1.3 implementation can still fail during the handshake. Amazon MSK does not automatically retry that failed connection using TLS 1.2.
Prerequisites Before Enabling TLS 1.3
Before changing the security policy, verify the following requirements.
Check cluster compatibility
TLS security policies are available for Amazon MSK Provisioned clusters, not MSK Serverless.
The cluster must also run Apache Kafka version 3.6 or later.
Additionally, the cluster must be in the ACTIVE state, with no other cluster operation running.
Review Kafka client compatibility
Identify every application that connects to the cluster.
This includes:
Java-based Kafka producers and consumers.
Python Kafka applications.
.NET applications using Kafka client libraries.
Kafka Connect workers.
Kafka administration tools.
Monitoring and observability applications.
Custom integration services.
Check whether the underlying TLS libraries support TLS 1.3 correctly.
For Java applications, review the installed JDK version and its TLS implementation. Also check whether the application or JVM explicitly restricts the supported protocol versions.
Check network intermediaries
TLS connections may pass through security appliances, proxies, firewalls, or traffic inspection systems.
These components must support TLS 1.3 correctly.
A Kafka client can have full TLS 1.3 support and still fail to connect if an intermediary interferes with the handshake.
Step 1: Identify the Current Amazon MSK Configuration
Start by retrieving the cluster configuration.
aws kafka describe-cluster \
--cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/example-cluster/example-id
Review the response for the following information:
Next, retrieve the configuration revision:
aws kafka describe-configuration-revision \
--arn CONFIGURATION_ARN \
--revision 1
Replace the configuration ARN and revision with the actual values associated with your cluster.
If the TLS security policy is not configured, Amazon MSK uses MSK-TLS1.2-Default-2026.
Before creating a new configuration revision, preserve the existing broker configuration properties. A new revision replaces the configuration applied to the cluster.
Step 2: Enable TLS 1.2 and TLS 1.3 Together
The safest migration approach is to introduce TLS 1.3 without immediately removing TLS 1.2 support.
Create a file named server.properties:
tls.security.policy=MSK-TLSv1.3-2026
If your existing configuration contains other broker properties, retain them in this file.
For example:
auto.create.topics.enable=false
default.replication.factor=3
min.insync.replicas=2
tls.security.policy=MSK-TLSv1.3-2026
The additional properties are illustrative. Use the values already approved for your environment rather than replacing existing settings blindly.
Create a new MSK configuration:
aws kafka create-configuration \
--name msk-tls13-compatible \
--kafka-versions 3.8.x \
--server-properties fileb://server.properties
Replace the example Kafka version with the version supported by your cluster.
The response includes the configuration ARN and revision number.
Next, retrieve the current cluster version:
aws kafka describe-cluster \
--cluster-arn CLUSTER_ARN \
--query "ClusterInfo.CurrentVersion" \
--output text
Apply the new configuration:
aws kafka update-cluster-configuration \
--cluster-arn CLUSTER_ARN \
--current-version CURRENT_CLUSTER_VERSION \
--configuration-info '{
"Arn": "CONFIGURATION_ARN",
"Revision": 1
}'
Amazon MSK performs a rolling restart to apply the policy.
The cluster continues serving traffic during the restart when its replication and availability settings meet AWS requirements, including an appropriate replication factor, min.insync.replicas, and client retry configuration.
Monitor the cluster operation until it completes.
Step 3: Test TLS 1.2 and TLS 1.3 Connections
After applying the compatibility policy, test both protocol versions independently.
You can use OpenSSL to verify the broker's TLS handshake.
First, test TLS 1.2:
openssl s_client \
-connect BROKER_ENDPOINT:9094 \
-tls1_2 \
-servername BROKER_ENDPOINT
Next, test TLS 1.3:
openssl s_client \
-connect BROKER_ENDPOINT:9094 \
-tls1_3 \
-servername BROKER_ENDPOINT
Use the correct broker endpoint and listener port for your authentication configuration. Port 9094 is commonly used for TLS client connections with certificate-based authentication.
A successful handshake should display the negotiated TLS version and cipher information.
For TLS 1.3, Amazon MSK supports these cipher suites:
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
Amazon MSK does not support TLS_CHACHA20_POLY1305_SHA256.
These OpenSSL checks verify the TLS handshake, not the complete Kafka authentication or authorization process. Follow them with actual producer and consumer connection tests.
Step 4: Update Existing Kafka Clients
Once the cluster accepts both protocols, begin upgrading applications that still use older TLS configurations.
For Java applications, review the JVM configuration and Kafka client settings.
For example, you can explicitly configure the JVM to support both versions:
java \
-Djdk.tls.client.protocols=TLSv1.2,TLSv1.3 \
-jar kafka-consumer.jar
This setting allows the Java TLS client to use the specified protocols, subject to JDK support and application configuration.
Do not introduce this setting without testing the application. Some applications may use custom SSL contexts or explicitly configured protocol restrictions.
For applications using other languages, verify that their TLS libraries and Kafka client dependencies support TLS 1.3.
During migration, keep track of which applications have been upgraded and which still depend on TLS 1.2.
Step 5: Move to TLS 1.3 Only
After confirming that all clients, tools, and network intermediaries support TLS 1.3, you can consider enforcing TLS 1.3 exclusively.
Update the configuration property:
tls.security.policy=MSK-TLSv1.3-Only-2026
Create a new configuration revision:
aws kafka update-configuration \
--arn CONFIGURATION_ARN \
--server-properties fileb://server.properties
Apply the returned revision to the cluster using update-cluster-configuration.
There is an important migration restriction.
Existing clusters must move between adjacent security policies:
MSK-TLS1.2-Default-2026 → MSK-TLSv1.3-2026 → MSK-TLSv1.3-Only-2026
You cannot move directly from the default policy to TLS 1.3 only in one operation.
Wait for the compatibility policy update to complete before applying the stricter policy.
Comparison of TLS Migration Approaches
Approach | Compatibility | Migration risk | Recommended use |
|---|
TLS 1.2 only | Older clients supported | Low | Existing environments |
TLS 1.2 and TLS 1.3 | Both client types supported | Moderate | Gradual migration |
TLS 1.3 only | TLS 1.3 clients required | Higher | Fully upgraded environments |
The dual-protocol policy is useful because it separates the initial protocol upgrade from the final enforcement decision.
However, it should not be considered a guarantee that every older client will work. Clients with incorrect TLS 1.3 implementations can fail even when TLS 1.2 remains enabled.
Common TLS 1.3 Migration Problems
Older Kafka clients cannot connect
An older client might not support TLS 1.3 or may use an outdated TLS library.
Check the client runtime and TLS implementation. If the client supports only TLS 1.2, it can continue working under the compatibility policy.
TLS handshake fails unexpectedly
A client may advertise TLS 1.3 but fail to complete the handshake.
Check its TLS implementation, supported cipher suites, and any intermediate network devices.
No shared cipher suite is available
Amazon MSK supports specific cipher suites for TLS 1.3.
If a client offers only an unsupported cipher suite, the handshake cannot complete. Configure the client to offer a supported AES-GCM cipher suite.
Cluster configuration update fails
Check that the cluster is ACTIVE, that no other operation is running, and that the target policy is the next policy in the migration sequence.
Also verify that the configuration contains the complete set of required broker properties.
Rolling Back a TLS 1.3 Configuration
If applications experience connection failures after enabling TLS 1.3, you can return to the previous compatible policy.
For example, if the cluster is running MSK-TLSv1.3-Only-2026, apply MSK-TLSv1.3-2026 to restore TLS 1.2 compatibility.
You do not need to return to TLS 1.2 only.
If the cluster is already using MSK-TLSv1.3-2026, you can return to MSK-TLS1.2-Default-2026 through a new configuration revision.
Remember that the migration order applies in both directions. A rollback from TLS 1.3 only to TLS 1.2 only requires two separate updates.
Keep the previous configuration revision and cluster details available as part of your change-management process.
Best Practices for Production Amazon MSK Clusters
Inventory every Kafka client before changing the cluster policy.
Test TLS 1.3 against a non-production MSK cluster first.
Use the dual-protocol policy during the initial migration.
Upgrade Java runtimes and TLS libraries before enforcing TLS 1.3.
Test producers, consumers, Kafka Connect workers, and monitoring tools separately.
Verify network intermediaries support TLS 1.3.
Monitor authentication errors, connection failures, and consumer lag during the rolling restart.
Maintain sufficient Kafka replication and configure appropriate client retries.
Keep the previous configuration revision for rollback planning.
Enforce TLS 1.3 only after every required client has passed compatibility testing.
Advantages and Disadvantages of TLS 1.3 in Amazon MSK
Advantages
Provides modern encryption for Kafka communication.
Reduces the TLS handshake round trips compared with TLS 1.2.
Supports modern cryptographic cipher suites.
Allows gradual migration using the dual-protocol policy.
Helps organizations meet security requirements that mandate TLS 1.3.
Disadvantages
Older clients may require runtime or library upgrades.
Incorrect TLS 1.3 implementations can cause handshake failures.
Network security appliances may require compatibility updates.
Migration requires careful configuration management and testing.
Enforcing TLS 1.3 only prevents TLS 1.2 clients from connecting.
Summary
Enabling TLS 1.3 for Amazon MSK is an important step toward modernizing Kafka communication security. However, the migration should be planned around client compatibility rather than treating it as a simple broker configuration change.
Amazon MSK provides three security policies, allowing administrators to start with TLS 1.2 and TLS 1.3 compatibility before moving to TLS 1.3 exclusively.
The recommended approach is to inventory existing clients, enable the dual-protocol policy, test application connections, upgrade older TLS implementations, and enforce TLS 1.3 only after successful compatibility validation.
A gradual migration reduces the risk of unexpected connection failures while helping organizations improve the security of their event-driven applications.
Join the conversation! Your thoughts help the community grow.