TopolearnKafka Streams turns topics into continuously updated results without a separate processing cluster.
Kafka Streams is a Java library (part of Apache Kafka) for processing topics. An application is a normal JVM process; you scale it by running more instances with the same application.id, which acts as the consumer group.
You describe a topology with the DSL:
- KStream: a stream of independent events (every order)
- KTable: a changelog view, the latest value per key (current customer profile); a compacted topic maps naturally to a KTable
- stateless operations:
filter,mapValues,flatMap,branch - stateful operations:
groupByKey().count(),aggregate,reduce, and windowed aggregations (tumbling, hopping, sliding, session) - joins: stream-stream (within a time window), stream-table (enrich events with current state), table-table
State lives in local state stores (RocksDB by default), each backed by a compacted changelog topic, so a failed instance's state is rebuilt elsewhere. Repartitioning (after changing the key) happens through internal topics automatically.
Set processing.guarantee=exactly_once_v2 for exactly-once results. Interactive queries let you read state stores directly from the app. Other options for stream processing include Apache Flink (separate cluster, very powerful) and ksqlDB-style SQL engines.
Example
import java.util.Properties;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
public class OrderCounts {
public static void main(String[] args) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-counts");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> orders = builder.stream("orders"); // key = customer
KTable<String, Long> counts = orders.groupByKey().count(); // state store + changelog
counts.toStream().to("orders-per-customer", Produced.with(Serdes.String(), Serdes.Long()));
KafkaStreams app = new KafkaStreams(builder.build(), props);
Runtime.getRuntime().addShutdownHook(new Thread(app::close));
app.start();
}
}Remember
- KStream = events, KTable = latest value per key
- Stateful ops use local RocksDB stores backed by changelog topics
- Scale by running more instances with the same application.id
- exactly_once_v2 gives exactly-once results within Kafka
Watch out for
- Changing the key and then aggregating without realizing a repartition topic is created, doubling traffic.