Data pipelines are infrastructure, and infrastructure is judged on the days it fails. Most estates we inherit fail the same ways: a job runs on schedule regardless of whether its input arrived, a partial file is loaded as though complete, a retry duplicates rows, a schema change breaks a transformation silently, and nobody finds out until a report looks wrong the following week.
This covers the patterns we apply across roughly 2,400 production DAGs, with Dataproc as the processing tier where Spark is warranted.
The five properties every pipeline needs
Idempotency first. A pipeline must produce the same result whether it runs once or five times. In practice: writes are merges keyed on a business identifier rather than appends, partitions are overwritten atomically rather than added to, and any external side effect is guarded by a deduplication key. Without this, retries are dangerous and backfills are terrifying — which is why so many organisations avoid both and accumulate permanent gaps.
Then dependency on data rather than time. Then observability with row-level metrics on every run. Then reconciliation against source for anything with financial or operational weight. And then ownership — a named person, an SLA appropriate to the business use, and an alert route that reaches someone who can act.
In practice
Sensors instead of hopeful schedules
A job scheduled at 03:00 because the source file usually lands by 02:30 will eventually process yesterday's file, and nobody will notice for a week. Sensors wait for actual data availability with a timeout that raises an alert rather than allowing the DAG to proceed on missing input.
This single change eliminates a whole category of silent data quality incident — the kind where the pipeline reports success and the numbers are simply stale. It is also the change most commonly missing from estates we take over.
Keep the DAG file itself light. Airflow parses DAG files constantly, so a database call or heavy computation at module level degrades the entire scheduler, not just that pipeline.
Ephemeral Dataproc clusters
A long-running Spark cluster is a standing cost that is idle most of the time. Ephemeral clusters — created per job, sized for that job, destroyed on completion — are both cheaper and cleaner, because every run starts from a known configuration rather than inheriting whatever the last job left behind.
Airflow orchestrates this natively: create cluster, submit job, delete cluster, with the delete running even on failure so a crashed job does not leave an expensive cluster running over a weekend. Preemptible secondary workers cut cost substantially for interruptible batch, with the primary workers on-demand for stability.
Size deliberately. The most common waste we see is a cluster sized for the largest job in the estate running every job, and the second most common is a job that could run on a single machine with Polars or DuckDB running on a cluster because Spark was the default.
| Pattern | Cost effect | Operational effect |
|---|---|---|
| Long-running shared cluster | Pays for idle time | Configuration drift between jobs |
| Ephemeral per-job cluster | Pays only for run duration | Known state every run |
| Preemptible secondary workers | 40–70% cheaper on batch | Needs checkpointing tolerance |
| Autoscaling policy | Matches spend to workload | Requires sensible min/max bounds |
| Single-machine engine for small jobs | Order-of-magnitude cheaper | One less system to operate |
Backfills as a first-class operation
Every estate eventually needs to reprocess history — a logic error found, a new field added, a source corrected. If backfill is not a supported operation, that reprocessing either does not happen or happens through a hand-written script under pressure.
The design property that makes backfill safe is idempotency, and the operational property is partitioning the work: run at controlled parallelism so neither the source nor the warehouse is overwhelmed, checkpoint progress so an interruption resumes rather than restarts, and reconcile counts as it proceeds.
We have backfilled several years of plant data at billions of rows this way. It is uneventful precisely because the pipelines were built to tolerate it from the start.
Observability beyond pass and fail
Green does not mean correct. A pipeline that succeeds while loading four hundred rows into a table that normally receives forty thousand has failed in the way that matters.
Record rows in, rows out, rows rejected, duration and code version on every run. Compare against historical norms and alert on anomalies. Track duration trends, because a task growing five per cent a week will breach its window within a couple of months and seeing that early is considerably cheaper than discovering it at 6 AM on a reporting day.
And tune alert severity. An estate that pages for everything trains people to ignore alerts, which is worse than having none.
Set an SLA per pipeline, not per estate
A daily management report has an SLA of "complete by 07:00, alert if not". A control-room feed might be five minutes end to end. Setting it explicitly forces the useful conversation about what actually matters, and it makes alerting meaningful rather than noise.
The operating model around the code
A pipeline estate needs an operating model as much as it needs code. Ownership per pipeline. An escalation route that reaches a human. Runbooks for every failure mode you can anticipate — what the alert means, how to diagnose, what to check, and whether the correct action is retry, skip, backfill or escalate.
That last artefact is what allows a client's own team to operate the platform overnight without needing the engineer who built it, and it is the difference between a handover that works and one that quietly returns to you six months later.
Key takeaways
- Idempotency is the property that makes retries and backfills safe — without it, both get avoided.
- Data-availability sensors eliminate the silent staleness failure that clock schedules produce.
- Ephemeral Dataproc clusters are cheaper and produce a known state on every run.
- Green does not mean correct — record row-level metrics and alert on anomalies against history.
- Runbooks are what let a client operate the estate overnight without the original engineer.
Frequently asked
ELT for analytics in most cases — land raw data first, transform in the warehouse with dbt. It preserves the source record, makes logic changes replayable without re-extraction, and puts transformation in SQL analysts can read. ETL still wins where data must be filtered or masked before landing for compliance reasons, or where volumes make transforming in flight cheaper than storing raw.
Airflow for most clients — the ecosystem, operator library and available talent are considerably deeper, and managed offerings exist on every cloud. Dagster has a better developer experience and an asset-oriented model, which we use where a team is starting fresh and values that. The deciding factor is usually who will operate it in three years.
Managed, unless you have platform engineering capacity or an on-premise requirement. Scheduler tuning, worker autoscaling, database maintenance and version upgrades are non-trivial, and the managed premium is usually less than the cost of doing it well yourself.
Usually, and it is often the right answer. The recurring problems are non-idempotent tasks, clock-based scheduling, no SLAs, no ownership and unmonitored alerting — all fixable incrementally without a migration. We start with an audit and give you a ranked list; several clients have executed it themselves.