Cloud, Data, Analytics & BI expert experienced in helping organizations gain positive business outcomes through the use of modern analytics. Pre-sales architect with over 18 years of experience assisting software sales in large enterprise sales cycles by creating innovative demos, developing prototype applications and creating solutions to address complex data related challenges in the field of business intelligence(BI) & Big data analytic & business process automation, imaging and workflow.

Comparative Analysis: Snowflake Dynamic Tables(DTs) vs. the new Databricks Declarative Pipelines (aka DLTs)

I recently completed a head-to-head performance test simulating Change Data Capture (CDC) pipelines across Bronze, Silver, and Gold layers. The goal was to evaluate how Snowflake Dynamic Tables compare to Databricks Declarative pipelines in terms of efficiency, cost, and ease of use.

Methodology
The test environment used identical TPCH-SF10 tables, with the largest containing 600 million rows. I executed four batches of identical CDC changes with three-minute pauses between each. The first two batches involved heavy inserts and updates across all three source tables. The final two batches were lighter, targeting only a single table to test the "intelligence" of each platform's refresh logic.

Key Performance Findings

- Cost & Speed: Snowflake outperformed Databricks significantly. Using a single X-Small Gen2 warehouse for all nine Dynamic Tables, Snowflake was 3 to 5 times cheaper while maintaining speeds twice as fast as the competition.

- Refresh Intelligence: Dynamic Tables demonstrated superior logic by identifying which specific tables required a refresh. In contrast, Databricks refreshed the entire pipeline even when only a single table had changed.

- Compute Control: Snowflake provides granular control over compute resources. Users can assign different warehouse sizes for initial full refreshes versus incremental updates. Databricks automates compute management, offering no control over specific pipeline SLAs.

- Configuration Complexity: The Snowflake setup was streamlined using standard SQL. Databricks required more manual work as in navigating the Pipeline UI, splitting code into multiple SQL files, enabling specific table settings that are not on by default and choosing the proper DLT method to avoid limitations like no support for Iceberg/Uniform (listed in my blog).

- Serverless Reality: The test highlighted not all "Serverless" are created equal. There was a distinct difference in serverless architecture. Snowflake pipelines start and stop in under 1 second with full control over compute. The Databricks serverless model is a managed Spark cluster that you can't control neither the size or performance, where you have to PAY EXTRA to get faster start times of ~60 seconds(still 1 min slower than Snowflake) unless you can wait for 10 mins for a job to start executing via standard-mode.

Conclusion
While Databricks added improvements in efficiency, such as Performance Mode and shared serverless clusters, the path to an optimized pipeline was notably more complex. 

For this specific CDC use case, Snowflake delivered a more cost-effective and user-friendly experience.

Most of these are documented in my latest blog about the test process, #architecture, Databricks learnings & limitations
https://lnkd.in/em_GHxUP

Git Repo for the code:
https://lnkd.in/ehf8QECw

taplio

Nick Akincilar's Linkedin Analytics

Nick Akincilar

Performance ScoreNick Akincilar's LinkedIn Performance

What is Nick talking about?

Nick Akincilar's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Nick Akincilar's Linkedin Analytics

Nick Akincilar

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Nick Akincilar's LinkedIn Performance

What is Nick talking about?

Nick Akincilar's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Nick Akincilar's Linkedin Analytics

Performance ScoreNick Akincilar's LinkedIn Performance

Nick Akincilar's Best Posts (last 30 days)