Data engineering for companies whose data has outgrown Excel

Our default architecture is a medallion lakehouse on open table formats — Apache Iceberg or Delta Lake — with three layers that each have a clear contract.

Bronze holds raw ingested data, immutable and exactly as received, with source metadata and ingestion timestamps. Nothing is cleaned here. This layer exists so that when a downstream calculation is questioned six months later, you can prove what the source actually said, and so that a logic change can be replayed from history rather than re-extracted from a system that may no longer hold it.

Silver holds conformed data: typed, deduplicated, with business keys resolved and slowly changing dimensions handled. This is where a plant tag becomes a named measurement with units, where a customer in the ERP is reconciled with the same customer in the CRM, and where quality rules are enforced.

Gold holds business-facing marts, modelled for the questions people actually ask — production by product by shift, cost per tonne, customer profitability, availability and utilisation. These are the tables BI tools and analysts touch, and they are designed for query performance and comprehensibility rather than normalisation purity.

Open table formats matter because they decouple storage from compute. Your data sits in your own object storage in an open format that Spark, Trino, Snowflake, Databricks, Athena and DuckDB can all read. You are not locked into a vendor whose pricing you will renegotiate under duress in three years.

Streaming is genuinely harder to build and operate than batch, and a great deal of it is built for reasons that do not survive scrutiny. The honest test is whether a decision changes because the data arrived in seconds rather than hours. A control-room alert on kiln temperature deviation: yes. A monthly management report: no.

Where it does earn its place we build on Kafka for durable, replayable ingestion and Spark Structured Streaming for processing, with the details that separate a demo from a production system: watermarking for late and out-of-order events, exactly-once semantics through idempotent writes and checkpointing, schema evolution handled without breaking consumers, dead-letter queues for records that fail validation, and backpressure that degrades gracefully rather than falling over.

For plant environments we usually pair this with a store-and-forward buffer at the edge, so a network interruption between plant and cloud never loses readings — they queue locally and back-fill automatically when the link returns. This is not an optional refinement; in Indian industrial settings the link will drop, and a system that loses data during those windows is a system nobody trusts.

Most value in most organisations still comes from well-run batch. We orchestrate with Apache Airflow — self-hosted, or on Cloud Composer, MWAA or Astronomer depending on your cloud — with a set of conventions that make hundreds of pipelines manageable.

Every DAG has an owner, an SLA, a retry policy with backoff, and an alerting route that reaches a human who can act. Tasks are idempotent so a rerun is always safe. Backfills are a first-class operation rather than a manual script. Dependencies are expressed as data availability rather than clock time, so a downstream job waits for its input instead of running at 3 AM and hoping.

We also treat pipeline observability as a deliverable. You should be able to see, for any table, when it last updated, how many rows it received, whether that is normal for this day of the week, and which upstream job produced it. Freshness and volume anomalies are where data quality problems announce themselves, usually days before anyone notices a wrong number in a report.

The fastest way to destroy trust in a data platform is to publish a number that is wrong once. After that every subsequent number is questioned, and the platform gets used as a starting point for a spreadsheet rather than as an answer.

So we enforce quality at the boundary. Every ingestion has a contract: expected schema, nullability, value ranges, uniqueness constraints, referential expectations and freshness thresholds. Records failing the contract go to quarantine with the specific reason attached rather than being silently dropped or, worse, silently coerced. The owning team gets an alert with the failing rows, and there is a defined path to correct and reprocess.

We also implement reconciliation as a routine operation: row counts and control totals compared between source and target for every critical pipeline, published as a report that finance and operations can see. This is unglamorous and it is the single practice that most reliably produces a platform people trust.

Cloud data platforms are priced in ways that punish careless design. A single badly written query scanning an unpartitioned table can cost more than a month of well-designed workload. We have taken over estates where 60% of the warehouse bill came from three scheduled queries that nobody had reviewed since they were written.

Our cost work is systematic. Partition and cluster tables on the columns people actually filter by. Compact small files, which are the silent killer of both cost and performance in lakehouses. Size warehouses and clusters to the workload with auto-suspend configured aggressively. Materialise expensive aggregations that are computed repeatedly. Move cold data to cheaper storage tiers with lifecycle policies. And attribute cost by team and by pipeline so the conversation about spend is specific rather than a general demand to reduce it.

Across recent migrations and optimisations the median reduction has been 41% of warehouse spend with no loss of capability — and usually with better query performance, because the same design choices that reduce scanned bytes also reduce latency.

Once a platform holds data from HR, finance and operations together, access control stops being optional. We implement role-based access at the table and, where needed, column and row level — so a plant manager sees their plant, a regional head sees their region, and salary columns are visible to a defined group only.

Lineage is captured automatically so any figure can be traced back through its transformations to the source records. This matters for two reasons: when someone challenges a number you can show the derivation, and when a source system changes you can see immediately what downstream reports are affected. We use catalogue tooling appropriate to the platform — Glue Data Catalog, Unity Catalog, or an open-source catalogue for self-managed estates.

For clients in regulated sectors we add retention policies, deletion workflows and access audit trails sufficient to satisfy India's data protection framework and, where relevant, customer contractual obligations.

Most engagements start with a two to three week assessment: what data exists, where it lives, what questions the business cannot currently answer, and what the current reporting actually costs in person-hours. That produces a target architecture and a phased plan, and you own that document whether or not you continue with us.

Build engagements then run in phases, each delivering a working slice — usually one subject area end to end, from ingestion through to a report someone actually uses. We resist the temptation to build the whole platform before delivering anything, because platforms built that way are frequently wrong in ways that only usage reveals.

Many clients then keep us on to operate the platform, either fully managed or as a support tier behind their own team. Data platforms are not a project that finishes; sources change, volumes grow, and the questions people ask evolve.