Topolearn

Learn Apache Kafka

Distributed event streaming platform: durable, replayable logs at massive scale

20 lessons · 4 hours · 4 levels · covers Apache Kafka 4.3 (KRaft)

Apache Kafka is a distributed event streaming platform that stores streams of records in replicated, partitioned, append-only logs that many applications can publish to and read from independently. It is the backbone of event-driven microservices, data pipelines, change data capture and real-time analytics at companies of every size. Learning it means learning to design systems around events instead of shared databases.

Sign in free to track progressOfficial docs ↗

Before you start

Apache Kafka builds on these; the tree links to the exact lessons you need.

PythonEssentials levelDockerEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics7 lessons · 1 hour in total
2. PracticalUse it productively at work, with the usual idioms12 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs17 lessons · 4 hours in total
4. ExpertInternals and the edge cases experts know20 lessons · 4 hours in total

The Apache Kafka tree

Apache KafkaWhat Kafka isWhat Kafka isTopics, partitions and offsetsTopics, partitions andoffsetsBrokers, replicas and the clusterBrokers, replicas andthe clusterRunning Kafka locally and the CLIRunning Kafka locallyand the CLIWriting a producerWriting a producerWriting a consumerWriting a consumerConsumer groups and scalingConsumer groupsand scalingKeys, partitioners and orderingKeys, partitionersand orderingDurable producers: acks and idempotenceDurable producers:acks and idempote…Offset commits and delivery guaranteesOffset commits anddelivery guaranteesSerialization and schemasSerialization andschemasRetention and log compactionRetention and logcompactionRunning containers with docker runRunning containerswith docker runDefining functionsDefining functionsfor and while loopsfor and while loopsExceptions in depthExceptions in depthDataclassesDataclasses
Essentials and Practical shown (12 of 20 lessons); the full list is below. Click a lesson to read it.

All 20 lessons, in order

Core concepts

  1. What Kafka is · Essentials · 8 min

    A distributed, durable log of events that producers append to and any number of consumers read independently.

  2. Topics, partitions and offsets · Essentials · 12 min

    Topics are split into ordered, append-only partitions; each record gets an offset, and ordering is guaranteed only within a partition.

  3. Brokers, replicas and the cluster · Essentials · 10 min

    A Kafka cluster is a set of brokers storing partition replicas, with one leader per partition and KRaft controllers managing metadata.

  4. Running Kafka locally and the CLI · Essentials · 12 min

    Start a single-node Kafka with Docker and use the bundled scripts to create topics, produce, consume and inspect groups.

Producers and consumers

  1. Writing a producer · Essentials · 12 min

    Send keyed records from Python with confluent-kafka: produce asynchronously, handle delivery reports and flush.

  2. Writing a consumer · Essentials · 12 min

    Read records with a poll loop: subscribe, handle messages and errors, choose a starting position and close cleanly.

  3. Consumer groups and scaling · Essentials · 12 min

    Consumers sharing a group.id split a topic's partitions between them; different groups each get all the data.

  4. Keys, partitioners and ordering · Practical · 12 min

    How producers choose partitions, why the hash function matters across clients, and how to deal with hot keys.

  5. Durable producers: acks and idempotence · Practical · 14 min

    Configure acks, retries, idempotence and min.insync.replicas so acknowledged writes survive broker failures without duplicates.

  6. Offset commits and delivery guarantees · Practical · 14 min

    Commit offsets manually after processing to get at-least-once delivery, and understand at-most-once and duplicates.

  7. Transactions and exactly-once processing · Advanced · 16 min

    Use transactional producers and read_committed consumers to atomically write outputs and commit input offsets.

  8. Rebalancing, liveness and group protocols · Advanced · 15 min

    Understand what triggers rebalances, tune poll and session timeouts, and use cooperative, static and next-generation group protocols.

Data, topics and retention

  1. Serialization and schemas · Practical · 14 min

    Encode record values as JSON, Avro or Protobuf, and use a schema registry to evolve event formats safely.

  2. Retention and log compaction · Practical · 12 min

    Control how long data stays with time and size retention, and keep the latest value per key with log compaction.

Streams and Connect

  1. Stream processing with Kafka Streams · Advanced · 18 min

    Build stateful stream processing apps in Java with KStream, KTable, aggregations, joins and windows, backed by Kafka.

  2. Kafka Connect and change data capture · Advanced · 15 min

    Move data between Kafka and databases, object stores and SaaS with Connect source and sink connectors, including Debezium CDC.

Performance and operations

  1. Throughput, latency and lag · Advanced · 15 min

    Tune batching, compression and fetch settings, size partitions, and monitor consumer lag and key broker metrics.

Reliability and internals

  1. Log storage internals · Expert · 15 min

    Segments, offset and time indexes, the page cache and zero-copy transfer: why Kafka's simple log design is so fast.

  2. ISR, high watermark and leader election · Expert · 16 min

    How followers replicate, how the ISR and high watermark define committed data, and what unclean leader election risks.

  3. KRaft and cluster metadata · Expert · 15 min

    How the KRaft controller quorum manages metadata through a replicated log, process roles, and running Kafka without ZooKeeper.