Streaming architecture is genuinely harder to build and operate than batch, and a substantial proportion of it is built for reasons that do not survive scrutiny. That is not an argument against streaming — we run Kafka and Spark Structured Streaming in production at twelve thousand events a second — it is an argument for applying it where it earns its complexity.
This piece is about how to tell the difference, and what it takes to run it properly once you have decided.
The test that decides it
Does a decision change because the data arrived in seconds rather than hours?
A control-room alert on a kiln temperature deviation: yes, clearly. Someone acts within the shift. A fraud check at the moment of transaction: yes. A stock reservation across sales channels so the same unit is not sold twice: yes. A monthly management report: no. A weekly sales analysis: no. A dashboard that senior management look at on Monday mornings: no, however much anyone wants it to be live.
Ask the question about the specific decision, not about the data. Plant tag data feeding a control-room alert warrants streaming; the same tag data feeding a monthly efficiency report does not, and both can come from the same ingestion with different consumers.
| Use case | Streaming? | Why |
|---|---|---|
| Control-room deviation alerting | Yes | Intervention happens within the shift |
| Cross-channel stock reservation | Yes | Prevents overselling and cancellation damage |
| Fraud and anomaly detection | Yes | The decision is at transaction time |
| Live operational dashboard | Usually | Only if someone acts on it during the shift |
| Daily production report | No | Batch at the cut-off is sufficient and simpler |
| Monthly management pack | No | Latency is irrelevant at this horizon |
Where streaming genuinely pays
In manufacturing, the clearest case is deviation alerting on process parameters. A kiln, a mill or a furnace drifting outside its band costs money every hour it continues, and an alert within minutes converts a fortnight of drift into a shift of drift. The ROI arithmetic is straightforward and usually compelling.
In commerce, cross-channel inventory reservation. Overselling causes cancellation, which damages marketplace ratings, which reduces visibility, which costs future sales — a compounding consequence from a fixable operational failure. Reservation must be immediate to work at all.
In logistics, geofenced arrival and departure events. Detention claims depend on a defensible record of when a vehicle arrived, and that record has to be created as the event happens rather than reconstructed later.
And across all of them, replayability rather than latency is frequently the real benefit. Kafka retains events, so a consumer that fails can resume, a new consumer can read history, and a logic error can be corrected by reprocessing rather than by accepting a permanent gap.
What separates a demo from production
Four things, and streaming systems that lose data in production are almost always missing at least one.
Watermarking with a documented late-arrival tolerance, agreed with the business, so late events are handled by a rule rather than dropped arbitrarily. Exactly-once semantics through idempotent sinks and checkpointed offsets — the phrase appears in a lot of design documents and is delivered by far fewer implementations. Bounded state, because unbounded aggregation state grows until the executor dies, usually at an inconvenient moment. And schema management through a registry with compatibility enforcement, so a producer change cannot silently break a consumer.
Add to that dead-letter handling with the failure reason attached and a defined replay path, plus backpressure that degrades gracefully rather than falling over.
In practice
The costs nobody quotes
Streaming infrastructure runs continuously, which means it costs continuously. A batch job that runs for twenty minutes a day costs twenty minutes of compute; a streaming job costs twenty-four hours. That is often fine and it should be in the business case rather than discovered in month two.
The larger cost is operational. Someone must monitor consumer lag, understand what rising lag means, and know what to do about it at 2 AM. Someone must handle a schema evolution that a producer team shipped without telling anyone. Someone must decide whether to replay from an offset after a logic fix. That is a genuine capability, and either you build it or you buy it.
For Indian mid-market clients we are frequently direct about this: if the requirement is genuinely batch, we do not install Kafka. The operational weight is real and it should be carried for a reason.
The question that saves money
Do multiple independent consumers need the same events, and do you need to replay history after fixing a consumer? If both answers are no, a managed queue such as SQS or RabbitMQ is simpler, cheaper and entirely adequate. Kafka earns its cost through fan-out and history, not through speed.
Micro-batch as the middle path
A great deal of what gets specified as streaming is actually satisfied by micro-batch — processing every one to five minutes. It delivers most of the operational benefit with considerably less complexity, because you keep batch semantics: a run either completes or does not, reprocessing is trivial, and there is no long-lived state to manage.
For an operational dashboard refreshed every two minutes, or an alerting rule evaluated every minute, micro-batch is very often the correct engineering trade. We propose it regularly and clients frequently prefer it once the operational difference is explained.
Reserve true streaming for cases where sub-minute latency genuinely changes the outcome, or where the event-by-event model is inherent to the problem — reservation, fraud, sequence-dependent processing.
A sequencing that works
Start by landing events durably. Get the producers writing to Kafka with sensible topic and partition design, with retention long enough to replay, and with edge buffering where plant links are involved. That alone provides value: a durable, replayable record that multiple consumers can read at their own pace.
Then add the consumer that has the clearest ROI — usually alerting, because the business case is easiest to demonstrate. Prove it, measure it, and let that fund the next one.
Then add batch consumers off the same stream for reporting and history, landing in the lakehouse. This is the point at which the architecture starts compounding: one ingestion, many consumers, no duplicate extraction from the source system.
Key takeaways
- The test is whether a decision changes with seconds versus hours — ask about the decision, not the data.
- Replayability is often the real benefit, more than latency.
- Watermarking, idempotent sinks, bounded state and a schema registry separate production from demo.
- Streaming costs run continuously and carry genuine operational weight — put both in the business case.
- Micro-batch is very often the correct middle path and should be proposed honestly.
Frequently asked
Ask two questions. Do multiple independent consumers need the same events? Do you need to replay history after fixing a consumer or changing logic? If both are no, a managed queue is simpler, cheaper and adequate. If either is yes, Kafka earns its operational cost. We have talked clients out of Kafka more than once.
Managed — MSK, Confluent Cloud or Event Hubs — unless data residency or very large scale forces otherwise. Kafka operations are genuinely specialised and the managed premium is usually less than building that capability in-house. For on-premise plant deployments we run self-managed with runbooks, because there is no alternative.
Consumer lag, without question. Rising lag is the earliest warning of a downstream problem, and estates that do not alert on it discover issues days later through a wrong report rather than through an alert.
Store-and-forward at the edge. Readings are written locally and forwarded when the link returns, back-filled automatically with original timestamps preserved. Reports mark back-filled periods explicitly so nobody is misled about when data actually arrived. In Indian plant environments the link will drop, and a system that loses data during those windows never earns trust.