Dave on Data

The Dave on Data mission is to empower any professional (or team) with the DIY data science skills needed to unleash the power of AI technologies like Copilot in Excel.

What gets left out of the AI Hype Cycle is this:

AI will make you (or your team) wildly more productive.

**BUT**

You must have the requisite skills to make the most of AI.

**AND**

If these skills are lacking, you will not be able to recognize (and correct) the AI when it makes mistakes.

BTW - AI makes mistakes all the time.

Take Microsoft's Copilot in Excel AI as an example.

The "magical" scenarios where Copilot analyzes data using a few clicks and a couple of short prompts rely heavily on two things:

Python
Machine learning

Here's the thing, however. 

I've tested Copilot in Excel, which rarely generates the correct Python/machine learning code.

As the DIY data scientist, this is where you unleash Copilot's power.

I can teach you these DIY data science skills.

Regardless of role. Regardless of background.

Founder

TDWI

I develop and deliver live training experiences on data science topics such as data analysis and predictive analytics (e.g., machine learning). I focus on accelerating the data science journey for teams and organizations using a practical hands-on approach.

Training can be delivered virtually or in person with your team. The following offerings are certified by TDWI and can be taught using R or Python:

👉 Machine Learning Bootcamp
👉 Visual Data Analysis
👉 Linear/Logistic Regresion

💻 Learn more: https://daveondata.com/live-analytics-training

Data Science Consultant & Instructor

Schedulicity

VP of Analytics

Data Science Dojo

VP of Data Science

Microsoft

Sr. Director, BI and Analytics

Principal Program Manager

Sr. Director, Platform Architecture

Principal Architect

Ascentium Corp

Associate Partner, Solution Architect

Safeco Insurance

IT Architect

e-software, Inc.

Software Design Engineer

Programmer Analyst

I help ANY professional or team have more impact using data. Analytics demystified.

Want to differentiate yourself at work with Excel skills? Leaders aren't excited about this in 2025:

- Using PivotTables
- Knowing many Excel functions

You know what they're excited about: AI

And that means Copilot in Excel.

Here are the 4 steps to build the skills you need to master Copilot in Excel and impress your manager, business stakeholders, and executives:

1) Ignore "prompt engineering."
2) Learn Python.
3) Learn DIY data science.
4) Recognize hallucinations.

I want to help you own the future of Microsoft Excel.

I will help you get started today - for free.

Join 1,000s of professionals building skills for the future: 
https://daveondata.com/newsletter

David Langer

As an author, there's nothing quite like waking up to some good news from Amazon!

Back in May of 2023, I saw an early version of Python in Excel for the first time. I immediately knew I wanted to be a part of the most significant upgrade in Excel's analytics capabilities ever. 

And it didn't stop there.

When I first got access to Copilot in Excel's "advanced analysis" mode, I quickly saw Microsoft's vision for Excel's future. For professionals building Python analytics skills, the future of Excel lies in deep integration with Copilot AI to unleash the power of their data.

My book is the first step in building the skills for Excel's AI-powered future. It uses your existing knowledge of Excel to help you learn the Python foundation you need - fast.

If you're ready to make 2026 the year you embrace the power of Excel, AI, and data, you can pre-order my book here: https://lnkd.in/eQp4g5_d

#Excel #PythonInExcel #MicrosoftExcel

I help professionals and teams build better forecasts using machine learning with Python and Python in Excel.

Interesting behavior from Copilot in Excel. Just another example of how Python in Excel skills are part of Excel's AI-powered future.

I was working on the next issue of my LinkedIn newsletter, which will cover using AI to build better forecasts in Microsoft Excel, and asked Copilot to create a forecast for me.

I was surprised to see that Copilot didn't use Excel's FORECAST.ETS() function because that's the best way to build data-driven forecasts using native Excel features.

Instead, Copilot in Excel generated Python code in the background to build an ETS model similar to what would be produced from FORECAST.ETS() using Python's mighty statsmodels library.

BTW - I covered statsmodels in my last two LinkedIn newsletter articles.

I'm betting that the new Copilot in Excel Agent Mode will be able to manipulate worksheets (e.g., generating formulas) directly and will be able to generate forecasts using FORECAST.ETS().

However, I'm also betting the Agent Mode will eventually be smart enough to recognize when a better forecasting technique is available from Python and use that instead.

Which means that the AI-powered future of Microsoft Excel will belong to those professionals who develop Python skills.

Stay healthy and happy data sleuthing!

#Excel #PythonInExcel #MicrosoftExcel

Wavicle Data Solutions

Lead Data Engineer

LinkedIn

LinkedIn Learning Instructor

Globant

Senior Data Engineer

CRISIL Limited

Senior Analyst - Data Engineer

Quantiphi

Data Engineer

Tata Consultancy Services

System Engineer

Assistant System Engineer

Assistant System Engineer - Trainee

InsideSherpa

KPMG Data Analytics Consulting Virtual Internship

Monisha Enterprises

•	Data mining and Extraction Transformation Loading using R programming.
•	Data cleaning and processing for Data analysis done using Python using NumPy, pandas, SciPy, Scikit. Data modelling and prediction was done using algorithms – Logistic Regression, Time Series Forecasting, Clustering and Classification.
•	Data Visualization done using Tableau for generating reports to enhance the productivity of the products.
•	Built a Data Warehouse on MSSQL Server for ERP System of the enterprise to organize the data for future processing.

Data Analyst Intern

Open to Collaborate | Storyteller | Lead Data Engineer@Wavicle| Linkedin Top Voice 2024 | Globant | Linkedin Learning Instructor | 2xGCP & AWS Certified | LICAP'2022

Data Engineer with 7+ years of experience working on data engineering, analysis, and modelling projects for various domains, such as banking, healthcare, sales, and e-commerce. I leverage my strong statistical and analytical ability to explore business ideas and provide innovative solutions. 

Skilled with technologies and strong intellect with enthusiasm as an avid learner with challenging opportunities. I’m working as a LinkedIn Learning Instructor to bring change as a data evangelist.

Post Graduated in Master of Technology in Data Science and Engineering from Birla Institute of Technology and Science, Pilani in 2023, where I learned advanced concepts and techniques in data science and engineering, machine learning, cloud computing, and visualization. 

I hold multiple certifications in GCP, AWS, and Azure, and I enjoy working with cloud technologies and big data platforms.

I am passionate about data science and women empowerment, and I have been recognized as the 
- Linkedin Top Voice 2024 
- Data Science and Women Empowerment Thought Leader by Global AI Hub. 

I am also a proud participant of the LinkedIn Creator Accelerator Program 2022, where I share my insights and experiences with the data science community. 

I am open to collaborate and share ideas, or explore various opportunities in the data field.

Pooja Jain

When everyone talks about data… but only one person actually fixes it. 

👉 It’s always the Data Engineer who climbs into the pit and makes the system work.

If you’re starting your journey as that person — the quiet builder behind every AI success — here’s a structured reading list and GitHub resources to build rock-solid foundations.

1. 𝗔𝗻𝗮𝗹𝘆𝘀𝘁 𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝘀
 • Tools: Excel, Power BI, SQL, Python
 • Focus: Automate reports, clean data, build dashboards
 • AI Boost: Use Copilot or ChatGPT to write Python scripts, generate SQL queries, and debug faster

2. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺
 • Tools: Pandas, dbt, Git, Jupyter
 • Focus: Write modular code, version control, transform data
 • AI Boost: Let AI help refactor messy code, explain Git workflows, and generate dbt models

3. 𝗗𝗮𝘁𝗮 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 
 • Tools: Airflow, Prefect, dbt, Databricks
 • Focus: Build ETL pipelines, schedule jobs, test transformations
 • AI Boost: Use AI to design pipeline architecture, write DAGs, and troubleshoot errors

4. 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗖𝗼𝗿𝗲
 • Tools: AWS/GCP/Azure, Spark, Kafka
 • Focus: Scale workflows, handle big data, deploy in cloud 
 • AI Boost: Use AI to generate infrastructure-as-code, optimize Spark jobs, and simulate Kafka streams

5. 𝗣𝗼𝗿𝘁𝗳𝗼𝗹𝗶𝗼 & 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀
 • Tools: GitHub, Streamlit, FastAPI, LinkedIn
 • Focus: Build end-to-end projects, document, share, and network
 • AI Boost: Generate project ideas, write documentation, and create interactive dashboards

But Will AI Replace Data Engineers❓
It’s a common fear. But here’s the truth:
AI won't take your job; 
it will just automate the boring parts, raising the floor for everyone else.

Explore these data engineering projects to upskill and level up-

Beginner:
ETL pipeline using Python and SQL by Ankit Bansal:
- https://lnkd.in/gTdCV9aJ
- https://lnkd.in/gxEYM3Bb

Intermediate:
- Data warehouse solution using Snowflake, dbt by Shashank Mishra 🇮🇳 : https://lnkd.in/gf-c5TR7
- Mr. K Talks Tech : https://lnkd.in/gt9JAkRt
- Snowflake project by Data Engineering Simplified : https://lnkd.in/gXRWyHpc
- Apache Spark project by Ankur Ranjan: https://lnkd.in/gt9JAkRt

Advanced: Implement a real-time streaming data processing
- Darshil Parmar : https://lnkd.in/ghiEsa7P
- Yusuf Ganiyu : https://lnkd.in/giYwJaCS

Cloud Projects:
- Microsoft Azure by Data Engineering Simplified - https://lnkd.in/gx3aqzKU
- Microsoft Azure by Sumit Mittal - https://lnkd.in/dX2yma5b
- Amazon Web Services (AWS) by Darshil Parmar - https://lnkd.in/gJg7KV-7
- Google Cloud by Anjan - https://lnkd.in/gjHbmCaM

System Design concepts with:
- Alex Xu - https://lnkd.in/gGdgJRDd
- Design Gurus - https://lnkd.in/gaphzp89

DataExpert.io handbook compiled by Zach Wilson -https://lnkd.in/gb4xBQJy
🔥 𝗔𝗜 𝘄𝗼𝗻'𝘁 𝗿𝗲𝗽𝗹𝗮𝗰𝗲 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀—𝗶𝘁 𝘄𝗶𝗹𝗹 𝗺𝗮𝗸𝗲 𝘁𝗵𝗲 𝗴𝗼𝗼𝗱 𝗼𝗻𝗲𝘀 𝘂𝗻𝘀𝘁𝗼𝗽𝗽𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗹𝗲𝗮𝘃𝗲 𝘁𝗵𝗲 𝗿𝗲𝘀𝘁 𝗯𝗲𝗵𝗶𝗻𝗱.

🏗️ Confused by Data Lake vs Warehouse vs Lakehouse vs Mesh?

Selecting the right architecture is game-changing for scalable, trustworthy analytics.

1️⃣ Data Lake
Think of a natural lake — it holds everything: clean water, mud, fish.
Similarly, a Data Lake stores raw data in any format. Flexible, but you’ll need to clean it before use.

2️⃣ Data Warehouse
Imagine a warehouse — everything is neatly packed and labeled.
A Data Warehouse stores structured, cleaned data ready for dashboards and reports. Reliable, but less flexible.

3️⃣ Data Lakehouse
Picture a house by the lake — you get the beauty of the lake and the comfort of a home.
A Data Lakehouse combines raw + structured data. Best of both worlds!

4️⃣ Data Mesh
Think of a network of homes — each family manages its own space.
Data Mesh decentralizes data ownership. Teams treat data as a product and manage it themselves.

🎯 THE SIMPLE TRUTH:
→ Starting out? Use a Data Warehouse (BigQuery, Snowflake, Redshift)
→ Need cheap storage for logs/raw data? Add a Data Lake
→ Want both? Modern Lakehouses (Databricks, Iceberg) are solid
→ Data Mesh? Don't even think about it until you're at massive scale

🎓 Major Takeaways from my experience:
→ Architectures must be aligned with specific project needs for optimal results.
→ Hybrid models like Lakehouse unlock flexibility and performance.
→ Evolving architectures foster innovation and reduce bottlenecks in data pipelines.

Embedding flexible, forward-thinking architecture is vital for data-driven success. 

Companies leading these transformations are able to pace up by 3-5x reducing infrastructure costs by 40-60% through right-sized, purpose-built data systems delivering insights.

Here are some resources to understand more about each of them -
→Designing Data-Intensive Applications by Martin Kleppmann - https://lnkd.in/drm6wQvH
→The Data Warehouse Toolkit by Ralph Kimball - https://lnkd.in/daWQwZ7Q
→Databricks Lakehouse Fundamentals (Free Course) - https://lnkd.in/gQEA-guy
→Google’s Data Mesh Playbook - https://lnkd.in/gumDAdfk

𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦 𝘪𝘴 𝘵𝘩𝘦 𝘣𝘢𝘤𝘬𝘣𝘰𝘯𝘦 𝘰𝘧 𝘪𝘯𝘯𝘰𝘷𝘢𝘵𝘪𝘰𝘯 𝘪𝘯 𝘥𝘢𝘵𝘢 𝘦𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨.

73% of data pipeline failures happen because of dependency issues, not bad code.
Cron jobs don't care if yesterday's data never arrived. They'll happily run on empty tables and produce garbage.

Airflow? It checks, retries, alerts, and refuses to cascade your failure downstream.

→ Beyond scheduling Airflow gives an important feature of 𝗗𝗔𝗚𝘀(𝗗𝗶𝗿𝗲𝗰𝘁 𝗔𝗰𝘆𝗰𝗹𝗶𝗰 𝗚𝗿𝗮𝗽𝗵𝘀), that helps to monitor, retry and gives flexibility to organize pipelines with minimum/less code.

🤔 Data Engineers often misinterpret Airflow just as an orchestration tool.

Apache Airflow is an open-source platform designed to programmatically author, schedule, retry and monitor workflows. 

🎼Think of it as the "conductor of an orchestra", where each musician (your data tasks, like data extraction, transformation, loading, or model training) knows when to play, what to play, and what happens if they make a mistake.

Most data engineers automate the wrong things at the wrong time.

Here's the framework I use after 8 years of building production systems:
✅ AUTOMATE WHEN: 
→ Task runs daily/weekly 
→ Human errors cause outages 
→ Work blocks other priorities
→ Team growth = more manual work
Examples: Reports, schema checks, alerts

❌ DON'T AUTOMATE WHEN: 
→ Task happens quarterly 
→ Requirements change weekly 
→ Process isn't understood yet 
→ Manual steps reveal insights

My rule: If it’s done 3+ times, script it; 10+ times, automate it; fails 5+ times, redesign it.
Automate what matters, when it matters—not everything!

Airflow isn’t just about running tasks.

🕑 It’s about building trustworthy, scalable, and observable pipelines that run reliably at 2am — even when you’re not watching.

Airflow gives structure to your chaos.
DAGs give clarity to your pipelines. 
That’s why DEs love them ❤️

Real Example: 
Instead of: 
❌ Cron jobs that fail silently 
❌ Manual dependency management
❌ No visibility into failures
You get: 
✅ Visual workflow monitoring 
✅ Automatic failure notifications
✅ Smart task scheduling 
✅ Easy debugging & restarting

Here's an amazing Airflow cheatsheet with some fundamentals concepts and interview questions to prepare for data engineers. 👇

Carousel Credits -  Futurense.com

Willing to learn more, here's the link to Astronomer free learning path - https://lnkd.in/g8-EWBda

Also, level up with Marc Lamberti's Udemy course on Airflow 3 - https://lnkd.in/gwArwaME

Airflow

Success and failure are separated by one decision: 
The choice to keep moving when everything says stop.

Agree?

Data governance is not the security guard saying “no”.

It is more like traffic lights in a busy city – a few rules that let everyone move faster and safely at scale.

Behind every confident decision is invisible governance doing the heavy lifting.
You know that feeling when you pull a report and immediately trust the numbers? 

When you can find the exact data you need in seconds? 
When your dashboard doesn't contradict itself?
That's not luck. That's governance.

🚫 The Misconception vs. ✅ The Reality

What People Think Data Governance Is:
 ❌ The "Department of No" that blocks data access
 ❌ Bureaucratic red tape that kills innovation
 ❌ A 500-page policy document gathering digital dust
 ❌ IT's way of controlling everything

What Data Governance Actually Is:
 ✅ The GPS system for your organization's data
 ✅ The quality control that makes "yes" possible
 ✅ The shared language that stops data arguments
 ✅ The trust layer that turns data into decisions

As a data engineer, you're not building barriers. You're building trust. 
You're not slowing things down. You're preventing the slowdowns that come from:
→ Wrong data in production
→ Compliance violations
→ Metric arguments
→ Security incidents
→ Data disasters

𝘎𝘰𝘰𝘥 𝘨𝘰𝘷𝘦𝘳𝘯𝘢𝘯𝘤𝘦 𝘪𝘴 𝘪𝘯𝘷𝘪𝘴𝘪𝘣𝘭𝘦. 𝘉𝘢𝘥 𝘨𝘰𝘷𝘦𝘳𝘯𝘢𝘯𝘤𝘦 𝘪𝘴 𝘪𝘮𝘱𝘰𝘴𝘴𝘪𝘣𝘭𝘦 𝘵𝘰 𝘪𝘨𝘯𝘰𝘳𝘦.

👉 Here's a quick Reality Check -
💵 Bad governance costs you:
→ 3 AM panic calls
→ Executive trust
→ Real money (hello, compliance fines!)
→ Your sanity

✅ Good governance gets you:
→ Predictable, reliable data
→ Business stakeholder love
→ Career advancement
→ Actual weekends

As a data engineer building governance always ensure that you track:
 • Time to Data - How long until new users can access data they need?
 • Data Trust Score - % of users who trust the data they use
 • Incident Response Time - How fast can you fix data quality issues?
 • Self-Service Adoption - Are users finding data themselves or asking you?
 • Compliance Coverage - % of sensitive data properly classified and protected
 • Metadata Completeness - How well documented is your data?

Here are some incredible folks to follow in the field of Data Governance -
Chad Sanderson
George Firican
🎯 Mark Freeman II
Piotr Czarnas
Dylan Anderson

𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝘀𝗻'𝘁 𝘁𝗵𝗲 𝗼𝗯𝘀𝘁𝗮𝗰𝗹𝗲 𝘁𝗼 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻—𝗶𝘁'𝘀 𝘁𝗵𝗲 𝗳𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝘁𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻 𝘀𝘂𝘀𝘁𝗮𝗶𝗻𝗮𝗯𝗹𝗲.

"I understand Git—I know git commit and git push."
That's like saying "I understand flying—I know takeoff and landing."

Everything in between? That's where pipelines break, data gets lost, and 3 AM incidents happen.

That the biggest myth in Data Engineering!
Even after 8+ years in tech, I've seen teams struggle not with writing code, but with managing it.

Git for Data Engineers: Understand Why It's Different
The Data Engineering Reality:

You're not just versioning code. You're versioning:
→ Pipelines that run on schedules
→ SQL transformations that power dashboards
→ DAGs that orchestrate entire workflows
→ dbt models that define business logic
→ Config files that control production behavior

𝘖𝘯𝘦 𝘸𝘳𝘰𝘯𝘨 𝘮𝘦𝘳𝘨𝘦 = 𝘗𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯 𝘥𝘰𝘸𝘯. 𝘋𝘢𝘴𝘩𝘣𝘰𝘢𝘳𝘥𝘴 𝘣𝘳𝘰𝘬𝘦𝘯. 𝘊𝘌𝘖 𝘢𝘴𝘬𝘪𝘯𝘨 𝘲𝘶𝘦𝘴𝘵𝘪𝘰𝘯𝘴.
Git isn't optional. It's your safety net.

🎓 Git Skills Every Data Engineer Needs

Level 1: Survival 
→ Clone, commit, push, pull
→ Create branches
→ Basic merges

Level 2: Proficiency  
→ Pull requests & code review
→ Resolve merge conflicts
→ Rebase vs merge (know the difference)
→ Cherry-pick commits

Level 3: Mastery 
→ Interactive rebase for clean history
→ Git hooks for automation
→ Submodules for shared code
→ Advanced conflict resolution

Here's a some resources to upskill with Git:
• Learn Git Branching - https://lnkd.in/dvPvYEeC
• Pro Git Book - https://lnkd.in/dasXAAYv

Without Git?
𝗬𝗼𝘂'𝗿𝗲 𝗳𝗹𝘆𝗶𝗻𝗴 𝗯𝗹𝗶𝗻𝗱, 𝗼𝗻𝗲 𝗺𝗶𝘀𝘁𝗮𝗸𝗲 𝗮𝘄𝗮𝘆 𝗳𝗿𝗼𝗺 𝗱𝗶𝘀𝗮𝘀𝘁𝗲𝗿.
With Git?
𝗬𝗼𝘂'𝗿𝗲 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗹𝗶𝗸𝗲 𝗮 𝗽𝗿𝗼, 𝘄𝗶𝘁𝗵 𝗰𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 𝗮𝗻𝗱 𝗰𝗼𝗻𝘁𝗿𝗼𝗹.

Image Credits: Priyanka Vergadia

What's your biggest Git fear or confusion as a data engineer? Let's solve it together in the comments.

taplio

The Battle

David Langer

Pooja Jain

Followers

Posts

Engagements

Likes

Performance ScoreDavid Langer's LinkedIn Performance

Performance ScorePooja Jain's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

David Langer

Pooja Jain

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}David Langer's LinkedIn Performance

Performance ScorePooja Jain's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreDavid Langer's LinkedIn Performance