Data · Since 2021 · Medallion estates in production

Databricks & Delta Lake

Lakehouse platform where Spark, streaming and machine learning share one governed layer.

Talk about a Databricks & Delta Lake project

Our position

Databricks is where we build when a client's workload genuinely spans SQL analytics, Spark transformation, streaming and machine learning, and they want one governed platform rather than four stitched together. Delta Lake underneath gives ACID transactions, schema enforcement, time travel and efficient merges on object storage — properties that turn a data lake from a file dump into something you can trust.

Our standard architecture is the medallion pattern: bronze holding raw ingested data exactly as received, silver holding conformed and validated data with business keys resolved, and gold holding the business-facing marts that BI tools and analysts touch. Each layer has a contract, and each is queryable by anything that reads Delta.

The platform is powerful and it is not cheap when used carelessly. Long-running all-purpose clusters, notebooks scheduled as production jobs, and unoptimised Delta tables account for most of the surprising bills we are asked to investigate.

Where it fits

Medallion lakehouse for multi-source industrial and commercial data

Structured Streaming from Kafka with exactly-once writes to Delta

Feature engineering and model training alongside the same data

Large-scale historical reprocessing and backfill

Organisations wanting one governed platform rather than several integrated ones

Why we choose it

What Databricks & Delta Lake genuinely gives us

One platform, four workloads

Batch, streaming, SQL and machine learning share the same data and the same governance instead of being separately integrated.

ACID on object storage

Delta gives transactional guarantees, schema enforcement and merge operations on cheap storage.

Time travel and versioning

Query a table as of last Tuesday, or restore after a bad load, without a backup restore.

Unity Catalog governance

Lineage, access control and audit across every asset in one place, which matters once HR, finance and operations data coexist.

Getting off notebooks without stopping delivery

The transition from notebook-driven analytics to engineered pipelines is where most Databricks estates either mature or stall. Nobody wants to hear that the pipeline producing the board pack is a notebook someone runs manually, but that is frequently the situation, and the risk sits with a single person.

We move these incrementally rather than declaring a rewrite. The logic inside a notebook is extracted into importable, tested modules while the notebook continues to run, calling into them. Once the module is trusted, the notebook becomes a thin caller, and then a scheduled job replaces it. At no point is the existing output interrupted, which is what makes the change politically possible.

Alongside that, the environment gets the discipline it lacked: code in a repository with review, deployment through CI rather than by copying cells between workspaces, job clusters instead of long-running interactive ones, and separate development and production catalogues so an experiment cannot overwrite a production table. Teams routinely find that the cluster cost falls at the same time, because interactive clusters left running are usually a large share of a Databricks bill.

Lakehouse in practice, once the demo is over

The lakehouse argument is sound: keep data in open formats in your own object storage, get transactional guarantees and time travel from a table layer such as Delta, and run whatever engine suits each workload. It avoids the historical trap of paying a warehouse vendor to hold data you cannot easily take elsewhere. What the demo does not show is that the discipline required is closer to a warehouse than to a data lake, and teams that arrive from a lake background usually underestimate this.

The failures we are called in to fix are consistent. Notebooks promoted to production, so critical pipelines exist as untested cells with no version control and no code review. The medallion structure adopted as folder names rather than as a contract, leaving a silver layer that is neither cleaned nor conformed. Small-file proliferation from streaming writes that nobody compacts, degrading every downstream read. And clusters configured once, generously, and never revisited.

Doing it properly is not complicated but it is deliberate: pipelines as tested code deployed from a repository, jobs run on job clusters rather than interactive ones, real contracts at each layer boundary with quality checks that fail loudly, table maintenance scheduled rather than hoped for, and Unity Catalog governance in place before the workspace fills with unowned tables.

Delta Lake table design, partitioning, Z-ordering and scheduled OPTIMIZE and VACUUM.

Notebook-to-production migration: version control, testing, CI and deployment from a repo.

Medallion architecture enforced as contracts with quality gates, not as folder naming.

Job cluster and pool configuration, spot usage and cost attribution by workload.

Unity Catalog governance, lineage and access control introduced before sprawl sets in.

The medallion layers and their contracts

Bronze is raw and immutable, with source metadata and ingestion timestamps. Nothing is cleaned here, because this is the layer that lets you prove what the source actually said when a figure is challenged eight months later, and that lets a logic change be replayed from history rather than re-extracted from a system that may no longer hold it.

Silver is conformed: typed, deduplicated, business keys resolved, slowly changing dimensions handled, quality rules enforced. Gold is modelled for the questions people actually ask, optimised for query performance and comprehensibility rather than normalisation purity.

Auto Loader for incremental file ingestion with schema evolution handling.

Delta MERGE for idempotent upserts so replays cannot duplicate.

OPTIMIZE and Z-ordering scheduled as routine maintenance, not run once at launch.

Expectations enforced at the silver boundary with quarantine rather than silent coercion.

Unity Catalog lineage so any gold figure traces to its bronze source.

Controlling Databricks cost

Job clusters rather than all-purpose clusters for scheduled work — all-purpose clusters left running are the most common source of unexpected spend. Aggressive auto-termination. Spot instances for interruptible batch with on-demand drivers. Photon where the workload benefits, measured rather than assumed. And small-file compaction, which affects both cost and query latency more than most teams realise.

Honest warnings

How Databricks & Delta Lake projects usually go wrong

These are the failure modes we look for first when we are called in to rescue somebody else's implementation.

Notebooks as production jobsHidden state, no tests, unpinned libraries. Convert to packaged modules with tests before scheduling.

All-purpose clusters for scheduled workMaterially more expensive than job clusters for identical output.

Never running OPTIMIZEDelta tables accumulate small files from streaming writes and read performance degrades steadily until someone notices.

Skipping Unity Catalog earlyRetrofitting governance across an established estate is far more work than enabling it at the start.

Straight answers

Databricks & Delta Lake questions we get asked

Including where we would recommend something other than Databricks & Delta Lake. Call +91 70033 91355 and you will get the same answer from an engineer.

+91 70033 91355 WhatsApp

Delta if you are on Databricks — the integration and tooling are tighter and the platform is built around it. Iceberg if you want maximum engine independence, since Spark, Trino, Snowflake, Athena, Flink and DuckDB all read it well. We use Iceberg for clients who want to keep their options open across query engines and Delta where Databricks is the committed platform. Both are open formats and both are good choices.

Often not. If your workload is purely SQL analytics, Snowflake or BigQuery will be simpler to operate and probably cheaper. Databricks earns its cost when Spark transformation, streaming and machine learning sit alongside SQL and you want them on one governed platform. We have recommended clients away from Databricks on exactly this basis.

Services built on Databricks & Delta Lake

Data EngineeringPetabyte-scale pipelines, lakehouses and warehouses — built by engineers who have run them in production for cement, steel, solar and commerce clients.ETL & Data PipelinesBatch and real-time pipelines that run every day without drama — orchestrated, monitored, idempotent, and honest about failure.Business Intelligence & AnalyticsDashboards people actually open, built on a modelled semantic layer so two departments never bring different numbers to the same meeting.AI Automation & Agentic AIAutonomous agents and AI systems that do real work inside your business — with guardrails, evaluation and a human in the loop where it matters.

Technologies we pair it with

Apache SparkDistributed processing for the workloads that genuinely need it — batch and streaming.PythonThe language our data, scraping and AI work is written in.Apache KafkaThe durable, replayable spine every real-time system we build sits on.Microsoft AzureThe right cloud when the organisation already runs on Microsoft.

Next step

Tell us what is slowing your business down.

A 30-minute call with a senior engineer — not a salesperson. You leave with an architecture sketch and an honest cost range, whether or not you hire us.

Book a discovery call