Apache Kafka
The durable, replayable spine every real-time system we build sits on.
Kafka is the component that makes real-time architecture possible rather than merely fashionable. Its value is not speed — plenty of things are fast — but durability and replayability. Events are retained, so a consumer that fails can resume, a new consumer can read history, and a logic error can be corrected by reprocessing rather than by accepting a permanent gap in the record.
We use it as the ingestion spine for plant telemetry, transaction events, clickstream and change data capture. Producers write once; any number of independent consumers read at their own pace without coordinating with each other or with the producer.
It also carries real operational weight, and we are direct about that. A Kafka cluster needs monitoring, capacity planning and someone who understands consumer lag. For clients whose requirement is genuinely batch, we do not install it.
Where it fits
What Apache Kafka genuinely gives us
Durable and replayable
Retention means a consumer failure is a delay rather than a data loss, and history can be reprocessed after a logic fix.
Decouples producers from consumers
A new consumer can be added without touching the producing system — which matters when the producer is a plant control system nobody wants to modify.
Ordering where it matters
Per-partition ordering guarantees let event-sourced and state-machine consumers be correct.
Scales past what most estates need
The clusters we run handle peaks well beyond design load without architectural change.
Topic and partition design decides everything downstream
A topic should represent one kind of event with one schema and one retention policy. Partitions should be keyed by whatever the consumers need ordering on — a machine identifier for plant data, a customer for transactions — because ordering is guaranteed within a partition and nowhere else.
Getting the key wrong is expensive to fix later, because repartitioning a live topic means reprocessing. We spend time on this at design and write it down with the reasoning.
Edge buffering for Indian plant networks
The link between a plant and a central cluster will drop. A system that loses readings during those windows never earns trust, so we place a store-and-forward buffer at the edge: readings are written locally, forwarded when the link is available, and back-filled automatically with their original timestamps.
The report or dashboard then shows explicitly which period was back-filled, so nobody is misled about when data actually arrived.
How Apache Kafka projects usually go wrong
These are the failure modes we look for first when we are called in to rescue somebody else's implementation.
Apache Kafka questions we get asked
Including where we would recommend something other than Apache Kafka. Call +91 70033 91355 and you will get the same answer from an engineer.
Ask two questions. Do multiple independent consumers need the same events? Do you need to replay history after fixing a consumer or changing logic? If both answers are no, a managed queue such as SQS or RabbitMQ is simpler, cheaper and entirely adequate. If either is yes, Kafka earns its operational cost. We have talked clients out of Kafka more than once.
Managed — MSK, Confluent Cloud or Azure Event Hubs — unless data residency or cost at very large scale forces otherwise. Kafka operations are genuinely specialised, and the managed premium is usually less than the cost of building that capability in-house. For on-premise plant deployments we run self-managed, with runbooks, because there is no alternative.
Services built on Apache Kafka
Technologies we pair it with
Tell us what is slowing your business down.
A 30-minute call with a senior engineer — not a salesperson. You leave with an architecture sketch and an honest cost range, whether or not you hire us.
Direct line
+91 70033 91355Mon–Sat · 9:30 AM – 7:30 PM IST · Sealdah, Kolkata