TopolearnMost integrations need no custom code: Connect gives scalable, fault-tolerant pipelines from configuration.
Kafka Connect is a framework, shipped with Apache Kafka, that runs connectors: plugins that copy data into Kafka (source connectors) or out of it (sink connectors). You run Connect as a cluster of workers in distributed mode and manage connectors through a REST API with JSON configs; Connect handles parallelism (tasks), offset tracking, retries and rebalancing for you.
Popular connectors include JDBC source and sink, S3 and GCS sinks, Elasticsearch and OpenSearch sinks, and Debezium for change data capture (CDC). Debezium reads a database's transaction log (PostgreSQL logical replication, MySQL binlog, and others) and emits every insert, update and delete as an event, with before and after images. CDC is the reliable way to stream database changes, unlike polling queries, which miss deletes and intermediate updates.
Other building blocks:
- converters (JSON, Avro, Protobuf) turn records into bytes, often with a schema registry
- single message transforms (SMTs) do light per-record changes: rename fields, route topics, extract keys
- dead letter queues (
errors.tolerance=allpluserrors.deadletterqueue.topic.name) capture bad records in sinks instead of stopping
The transactional outbox pattern combines CDC with your service's database: write events to an outbox table in the same transaction as the business change, and let Debezium publish them.
Example
# register a Debezium PostgreSQL source connector via the Connect REST API
curl -s -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" -d '{
"name": "shop-cdc",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "debezium",
"database.password": "secret",
"database.dbname": "shop",
"plugin.name": "pgoutput",
"topic.prefix": "shop",
"table.include.list": "public.orders"
}
}'
# changes to public.orders now appear in topic shop.public.orders
curl -s http://localhost:8083/connectors/shop-cdc/status
# {"name":"shop-cdc","connector":{"state":"RUNNING",...},"tasks":[{"state":"RUNNING",...}]}Remember
- Source connectors write into Kafka; sinks write out
- Connect manages tasks, offsets and failures; configure via REST
- Debezium CDC streams every row change from the database log
- Use converters, SMTs and dead letter queues
Watch out for
- Polling a table with a timestamp query and silently missing deletes; use log-based CDC.