Stream Processing and Batch Jobs
Turn many events into a useful answer
Follow the responsibility
A daily sales report can wait for a fixed set of records. A live fraud signal needs an answer as events arrive. Batch processing works on a bounded dataset; stream processing handles an ongoing sequence and often maintains changing results.
Neither approach removes late data or repeated events. Define what the answer means, when it is considered final and how a correction reaches the user.
What you’ll learn
- Compare bounded batches with ongoing streams.
- Distinguish event time from processing time.
- Plan windows, checkpoints and safe replay.
The event’s clock and the processor’s clock differ
Event time is when an event occurred according to its recorded timestamp. Processing time is when a processor handles it. An offline device can send an event minutes later. Counting only arrival time can put that event into the wrong business window.
A window groups events, such as sales within each ten-minute interval. A watermark estimates event-time progress; it is not proof that no earlier event will ever arrive. Define allowed lateness and whether late events revise an output, go to a repair path or are rejected.
State and output must survive a restart coherently
A running total needs state and a position in the input. Checkpoints can record progress and state so the processor resumes after failure. Replaying input may repeat output writes; use an output key, version or supported transaction mechanism to avoid double-counting.
A framework’s exactly-once claim has a boundary. Its state and broker integration may support coordinated recovery while an arbitrary external email or payment still repeats. Verify the complete input, state and sink contract.
Choose a correction and recovery rule
Try the design choices
Compare the visible consequences. With JavaScript enabled, choose an option in each scenario to assemble your decision record.
This is an exploratory design exercise. It does not run infrastructure, measure capacity or replace the lesson’s quiz.
Compare the user’s required freshness with the information available to the processor.
Choose processing from timing and state needs
Apache Kafka supplies a retained event log; Kafka Streams is a processing library for Kafka applications. Apache Flink supports stateful streaming and batch processing. Apache Spark supports large-scale batch work and streaming approaches. A scheduled SQL query or small worker may be enough for a simpler requirement.
Read Kafka Streams’ core concepts for its time and processing model. Measure events per second, state size, key skew, checkpoint costs, recovery time and sink capacity. A hot key or slow output can constrain the entire pipeline.
Use messaging for delivery and event contracts for the meaning of each record. Choose retained input and privacy rules before promising unrestricted replay.
Check your reasoning
Check your reasoning
Connect streams with history and query patterns
Compare independent event subscriptions, authoritative event history and derived query responsibilities. A retained stream does not automatically choose the application’s source of truth.
Bring the decisions together
Define the clock, window and correction rule. Connect input progress, processing state and output effects so recovery preserves the result’s meaning.
Reaching the recap and completing the quizzes are separate milestones.