An order changes status. Search indexes, operations dashboards, and risk monitoring all need to catch up. Having each team poll the application database duplicates work and leaves systems updating at different times. Adding message publishing to every application endpoint creates a different problem: background jobs and manual corrections are easy to miss.
This is a good job for a data synchronization tool.
Apache SeaTunnel is a high-performance, open-source distributed data integration tool. It connects databases, messaging systems, data warehouses, and search systems to read, lightly transform, and write data. For engineers, the appeal is practical: broad connectivity, efficient data movement, and synchronization jobs that can be managed as long-running workloads.
Its connectors cover familiar systems such as MySQL, PostgreSQL, Kafka, and Elasticsearch, as well as cloud services including Snowflake and Amazon S3. Instead of writing integration code for every new connection, teams can define a pipeline with Source, Transform, and Sink configuration, adding field mappings, filters, and lightweight processing along the way.
Efficiency and reliability matter just as much as connectivity. SeaTunnel supports distributed execution, with parallel reads for connectors that support splitting the source. Batch, streaming, and CDC cover different update patterns, while checkpoints preserve recoverable job state. Together, these capabilities shift the work from maintaining scattered data-moving scripts to designing a sustainable pipeline that downstream applications can actually use.
This article starts with a familiar combination: MySQL stores orders, and Kafka delivers order changes to multiple consumers. Rather than simply moving rows, we will standardize fields and preserve source metadata so consumers can tell where a record came from and what changed.
One pipeline, clear responsibilities
The MySQL CDC Source reads the existing orders and subsequent binlog changes. The Metadata Transform extracts the database, table, and row kind. The SQL Transform standardizes business fields. Finally, the Kafka Sink writes messages and places source metadata in their headers.
MySQL orders
↓ snapshot + binlog
MySQL-CDC Source
↓
Metadata → SQL Transform
↓
Kafka Sink
↓
Kafka topic → search / analytics / monitoring
This illustrates SeaTunnel's EtLT data pattern: apply the necessary lightweight transformations as data moves, so downstream systems receive something easier to use. Here, that means three specific operations, not complex analytics: rename id to order_id, map status codes to business names, and add source information.
Kafka decouples consumers. Search and analytics can advance their offsets independently. A new consumer can read messages within the retention window without requiring another extraction program against the application database.
Use the official deployment path
SeaTunnel provides official Docker and Kubernetes deployment instructions. Jobs run through the distribution's seatunnel.sh; there is no need to build a separate Python launcher.
With Docker, the workflow is straightforward: pull the image, mount a job configuration, and submit the job. This example pins version 3.0.0 rather than allowing latest to change the runtime:
docker pull apache/seatunnel:3.0.0
MySQL and Kafka must be reachable from inside the SeaTunnel container. The configuration below assumes an existing development environment on the Docker network seatunnel-network, with the network aliases mysql and kafka. Adjust the endpoints if your services use different addresses.
Enable row-based binlogging in MySQL and grant the CDC account the required read and replication permissions. Create the target Kafka topic before submitting the job. The MySQL CDC connector documentation details the database settings, permissions, and driver requirements. The official apache/seatunnel:3.0.0 image used here includes the MySQL CDC and Kafka connectors and the MySQL JDBC driver, so this job needs no additional installation. If your team builds a stripped-down image, include matching dependencies using the official plugin instructions.
Give consumers business fields, not internal status codes
The orders table has five fields:
Field | Meaning |
|---|
id
| Order primary key |
order_no
| Externally used order number |
user_id
| User identifier |
status
| Status code |
amount
| Order amount |
Consumers would rather receive order_id and status_name than repeat the mapping for codes 0, 1, and 2 in every application. SeaTunnel's SQL Transform performs that row-level transformation once, within the pipeline.
The Metadata Transform also exposes Database, Table, and RowKind as fields. The Kafka Sink then uses kafka_headers_fields to move them into headers, keeping the JSON payload focused on business data. “Move” matters here: these fields are removed from the JSON payload, not copied into both places.
The configuration follows the official MySQL CDC to Kafka recipe, retaining its Source → Metadata → SQL → Kafka processing flow and using local connection settings. This is a normal SeaTunnel job configuration, not another application to maintain.
Save the following as jobs/mysql-orders-to-kafka.conf. It reads lab.orders and writes to orders-cdc. Before running, replace every occurrence of REPLACE_WITH_CDC_PASSWORD with the same password chosen for this independent development environment. The setup appendix also uses REPLACE_WITH_ROOT_PASSWORD; replace that consistently with a separate local root password. These are placeholders, not credentials. For an existing database, use an appropriately authorized account.
env {
parallelism = 1
job.mode = "STREAMING"
}
source {
MySQL-CDC {
plugin_output = "mysql_orders_raw"
url = "jdbc:mysql://mysql:3306/lab"
username = "cdc"
password = "REPLACE_WITH_CDC_PASSWORD"
server-id = 61719
table-names = ["lab.orders"]
startup.mode = "initial"
schema-changes.enabled = false
}
}
transform {
Metadata {
plugin_input = "mysql_orders_raw"
plugin_output = "mysql_orders_with_meta"
metadata_fields {
Database = source_database
Table = source_table
RowKind = change_type
}
}
Sql {
plugin_input = "mysql_orders_with_meta"
plugin_output = "kafka_orders"
query = "select id as order_id, order_no, user_id, amount, case when status = 0 then 'CREATED' when status = 1 then 'PAID' when status = 2 then 'SHIPPED' else 'OTHER' end as status_name, source_database, source_table, change_type, CONCAT(source_database, '.', source_table) as source_name, 'mysql_cdc' as sync_source from dual where id is not null"
}
}
sink {
Kafka {
plugin_input = "kafka_orders"
bootstrap.servers = "kafka:9092"
topic = "orders-cdc"
format = json
partition_key_fields = ["order_id"]
kafka_headers_fields = ["source_database", "source_table", "change_type"]
}
}
Run this from the directory containing jobs:
docker run --rm -it \
--name seatunnel-orders \
--network seatunnel-network \
--memory 3g \
-v "$PWD/jobs:/config:ro" \
apache/seatunnel:3.0.0 \
./bin/seatunnel.sh -DJvmOption="-Xms512m -Xmx2g" \
-m local -c /config/mysql-orders-to-kafka.conf
This invokes the official entry point inside the image directly. If you do not already have MySQL and Kafka available, use the Docker commands in the Setup appendix below to create independent instances first.
Three points deserve attention during code review:
Preserve the business key. After renaming the field, the Kafka partition key must refer to order_id as well.
Preserve change semantics. Consumers use the change_type header to distinguish inserts, before-update rows, after-update rows, and deletes. The JSON business fields alone do not convey all of that information.
Agree on field responsibilities. source_database and source_table identify provenance; the order number, status, and amount drive business processing. Each consumer should not have to infer the contract independently.
Check the message contract, not just job status
A successful deployment is not the final check. First confirm that snapshot rows reach Kafka. Then update an order and insert another, checking both the business fields and change types.
For this JSON-plus-headers configuration, inspect the Kafka key, headers, and value separately. Printing only the value hides the metadata deliberately placed in headers. Do not interpret this contract as though it were another serialization format: JSON plus headers and debezium_json are not interchangeable.
A consumer maintaining the current state of orders needs an explicit processing contract. In this example, an order's id never changes:
On +I or +U, upsert by order_id, replacing the current state with the complete business fields in the message.
On -D, remove the state for that order_id.
On -U, ignore the before-update row; the following +U provides the new state. This convention applies to the fixed-primary-key, full-row-state scenario here. It does not automatically apply to primary-key changes or aggregations that must retract old values.
Apply changes in offset order within each partition. With a stable key and a fixed topic partition count, this run placed each order's records in the same partition. There is no global ordering across partitions, and configuring a partition key does not establish end-to-end exactly-once processing. Revisit ordering if you increase the partition count, change the key, or introduce concurrent processing that can reorder records. Primary-key upserts tolerate repeated writes of the same state; they do not prevent an older state from overwriting a newer one during out-of-order replay.
Consumers that charge a payment or send a notification also need their own business-level deduplication and transaction design. CDC captures database changes. It does not automatically turn every row change into a fully specified business event.
In the run using the official image, two initial orders produced two snapshot messages. Updating order 1001, inserting order 1003, and deleting order 1002 brought the total to six: -U and +U for the update, +I for the insert, and -D for the delete.
Order 1001 retained the key {"order_id":1001}. Its snapshot, before-update, and after-update records were all in the same partition. The updated payload contained status_name = SHIPPED and amount = 39.99, with provenance and change type in the headers. After applying the deletion of order 1002, the materialized current state contained only orders 1001 and 1003, matching the source table.
These results describe this JSON-plus-headers path, not every message format. Checking the key, headers, and value separately makes it easier to identify whether a problem lies in capture, transformation, or the consumer's interpretation.
From a local pipeline to a managed workload
Once the pipeline works, the reusable assets are its connection settings, field contract, and operating model, not a script that only runs on its author's laptop.
For long-running workloads, manage pinned connectors, drivers, and base configuration as part of the image. Inject database credentials through controlled configuration and secret-management mechanisms instead of leaving them in copied commands. Plan Kafka retention, MySQL binlog retention, and SeaTunnel recovery state together: they determine how far consumers can replay, where the Source can resume, and which job state can be restored.
Teams already using Kubernetes can follow the official deployment model with separate Master and Worker components. Prepare shared state storage, service discovery, resource limits, and monitoring, then submit the same data-processing logic. Containerization changes how the pipeline is delivered and operated; it does not require rewriting Source, Transform, and Sink as another program.
Less data-moving code, more usable data
MySQL to Kafka is more than reading a row and publishing a message. A useful pipeline also gives consumers a clear source, primary key, change type, and field contract.
SeaTunnel brings source capture, lightweight transformation, and destination writes into one configuration. The first pipeline delivers more than a data transfer: it establishes an integration approach that a team can review, deploy, and reuse. For engineers already working with MySQL and Kafka, that is a practical place to start.
Setup appendix: an independent Docker environment
These commands use the official seatunnel.sh entry point. They create an isolated local development setup, do not connect to a production database, and do not publish container ports to the host. If any resource name is already in use, choose another name and update the corresponding commands rather than deleting existing resources.
Create a working directory with a jobs subdirectory and save the configuration above as jobs/mysql-orders-to-kafka.conf. Run the commands from that working directory. Replace the two password placeholders consistently before running; the CDC password in the SQL below must match the job configuration. Use shell-safe values for these literal command examples, or quote and escape values appropriately.
Before creating the network, confirm that 10.243.173.0/24 does not overlap your local, VPN, or existing Docker networks. If it does, choose another unused private subnet. An exhausted Docker default address pool is not a reason to delete unrelated networks.
1. Create the network and MySQL source
docker network create --subnet 10.243.173.0/24 seatunnel-network
docker run -d --name seatunnel-mysql --network seatunnel-network --network-alias mysql \
--memory 1g -e MYSQL_ROOT_PASSWORD=REPLACE_WITH_ROOT_PASSWORD mysql:8.0.46 \
--server-id=1711 --log-bin=mysql-bin --binlog-format=ROW --binlog-row-image=FULL
Wait until the account can actually connect before initializing the table:
until docker exec -e MYSQL_PWD=REPLACE_WITH_ROOT_PASSWORD seatunnel-mysql mysql -uroot -e 'SELECT 1' >/dev/null 2>&1; do
sleep 3
done
If MySQL never becomes ready, inspect docker logs seatunnel-mysql rather than repeatedly executing initialization SQL. The account below is for this independent development environment only.
docker exec -i -e MYSQL_PWD=REPLACE_WITH_ROOT_PASSWORD seatunnel-mysql mysql -uroot <<'SQL'
CREATE DATABASE lab;
CREATE USER 'cdc'@'%' IDENTIFIED BY 'REPLACE_WITH_CDC_PASSWORD';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO 'cdc'@'%';
CREATE TABLE lab.orders (
id BIGINT NOT NULL PRIMARY KEY,
order_no VARCHAR(64) NOT NULL,
user_id BIGINT NOT NULL,
status INT NOT NULL,
amount DECIMAL(10,2) NOT NULL
);
INSERT INTO lab.orders VALUES (1001,'ORD-1001',501,0,19.99),(1002,'ORD-1002',502,1,29.99);
SQL
2. Start Kafka
docker run -d --name seatunnel-kafka --network seatunnel-network --network-alias kafka \
--memory 1500m \
-e KAFKA_NODE_ID=1 \
-e KAFKA_PROCESS_ROLES=broker,controller \
-e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
apache/kafka:3.9.1
Once the broker is ready, create a topic with two partitions:
docker exec seatunnel-kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server kafka:9092 --create --topic orders-cdc --partitions 2 --replication-factor 1
This is a single-broker development setup, not a highly available production deployment.
3. Submit the job through the official entry point
docker pull apache/seatunnel:3.0.0
docker run --rm -it --name seatunnel-orders --network seatunnel-network --memory 3g \
-v "$PWD/jobs:/config:ro" apache/seatunnel:3.0.0 \
./bin/seatunnel.sh -DJvmOption="-Xms512m -Xmx2g" -m local -c /config/mysql-orders-to-kafka.conf
Leave that terminal running. In a second terminal, inspect the messages:
docker exec seatunnel-kafka /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server kafka:9092 --topic orders-cdc --from-beginning \
--property print.key=true --property print.headers=true \
--property print.partition=true --property print.offset=true
4. Exercise incremental changes
docker exec -i -e MYSQL_PWD=REPLACE_WITH_ROOT_PASSWORD seatunnel-mysql mysql -uroot <<'SQL'
UPDATE lab.orders SET status=2,amount=39.99 WHERE id=1001;
INSERT INTO lab.orders VALUES(1003,'ORD-1003',503,0,59.99);
DELETE FROM lab.orders WHERE id=1002;
SQL
Inspect the change_type header: snapshot rows and inserts use +I, before-update and after-update rows use -U and +U, and deletes use -D. The updated order 1001 should contain SHIPPED and 39.99. Output from different partitions can interleave; use each partition's offsets to check ordering rather than treating terminal output as a global transaction sequence.
5. Stop the containers
docker stop seatunnel-orders seatunnel-mysql seatunnel-kafka
Stopping these containers does not delete the database data. There is no need to remove images, networks, or volumes to complete this walkthrough. Do not run a global prune command.
Official documentation
Join the conversation! Your thoughts help the community grow.