Kafka Window Stores clarification

Question

In my application, I defined a global state store (backed by a topic “query-topic”) in order to perform specific time based operations such as "give me all events in the query-topic from yesterday 8PM until today 5AM", or “give me all events in the query-topic for the last 3 days". I created the store using a window store builder as it seemed more efficient to execute time-ranged queries than a simple key value store.

Stores.windowStoreBuilder(
                Stores.persistentWindowStore(name, retentionPeriod, windowSize, retainDuplicates),
                Serdes.String(),
                valueSerde);

Nevertheless, the explanations regarding exactly how these window stores work is quite light. I couldn't find any relevant resources on Kafka official documentation and therefore had to rely on the Javadocs, which are not really explicit either. Moreover, I saw that another implementation called persistentTimestampedWindowStore also exists, which is a bit confusing for me because I thought that the WindowStore was already relying on Kafka event timestamp for the keys.

Could someone explain or redirect me to resources showcasing how such window stores work ? I can see that we can specify a retention period and window size, but how are these windows created ? When you receive a new record, do the windows move accordingly to this new value or are they current-time based ? How do range queries work when spanned over several windows ? I’m a bit lost.

Answer 1

I recommend you to start with, if you didn't yet, reviewing the time and windowing concepts using the official Kafka documentation as it goes straight to the point with some nice graphics to illustrate the concepts:

Windowing

Time

From there you might go to Confluent Developer series which has a lot of nice free resources like videos, code samples, articles. Official training is paid and expensive but their free material and code samples is extensive.

Confluent Developer

They also have some free ebooks.