DataOps.live

In this role, I excel at mentoring and educating customers regarding Amazon Web Services (AWS) and Azure configuration and installation of agent-based runner, adjusting to regulatory and security rules and ecosystems in finance, marketing, retail, healthcare, and e-commerce industries. I also develop best practices and manage platform projects utilizing Snowflake and support principal architects, data architects, data modelers, data engineers, the VP of IT infrastructure, and the Chief Information Officer (CIO). 
CORE ACCOMPLISHMENTS
•	Guided customers on how to consolidate data from various systems for greater data analyst palatability and usage.
•	Redirected and centered technical customers to focus on utilizing data structures and integration for business success. 
•	Exercised exceptional patience and demonstrated superb knowledge while guiding platform-training customers through month-long “handholding” onboarding phase with daily questions until self-sufficiency was forged; followed up accordingly.

Senior Solutions Architect

Team San Filippo

Chief Technical Advisor

EVERYTHING BUT THE HOUSE (EBTH)

Here, I automated and optimized business processes by instituting data science linear programming technique. Additionally, I utilized Python and operations research to assist with the optimization process. 
CORE ACCOMPLISHMENTS
•	Transformed process to automatically estimate item valuation based on historical auctions and selection of items presented for typical 7-day auctions. 
•	Reduced workload of merchandizing team dramatically, from 3 members spending 20 hours each weekend to just 30 minutes by single team member, freeing up valuable time for team to focus on more specialized auctions as well as estate sales.

Senior Data Scientist

Firstlight Media

I defined the Data Strategy for bringing data together from multiple applications to a consolidated enrichment platform that produces data visualizations. I designed and implemented the tools and techniques for producing consumer and content-based recommendations for our media player using graph technologies.

Data Strategist

KORE Wireless

I modernized our SQL Server environment to apply best practices in database use. 

I act as a Data Science Mentor, and Data Architect as we design new applications and use-cases for our data products. 

I implemented consolidated management of our SQL Server infrastructure using Idera tools. Upgraded our environment to current versions of SQL Server. 

I built an ETL team for managing the integrations that tie our business systems Netsuite and Salesforce to our internally developed applications. 

Using Signal analysis, I was able to demonstrate capabilities that could be implemented for any kind of anomaly detection as we monitor devices for our customers.

Senior Manager Data Services.

Cengage Learning

I created an ensemble predictive model using SQL rules, and R decision tree's to accurately (Precision/Recall both above 80%) predict the interactions with our platforms that at risk students exhibit. I created a Semantic analysis process for survey data to quickly parse through hundreds of feedback comments to focus customers on the issues and sentiment most important to respondents.

Data Scientist

Illumination Works

I managed our Data Science lab. Our Data Science Lab using both Map Reduce and Spark is based on CDH 5 from Cloudera. It is built using Cloudera Director and Cloudera Manager in AWS. Presentations built using our Data Science Lab were presented in December 2014 and January 2015.

Illumination Works is a solution enabling company. We provide professional consulting, design, and development services that deliver core information and insight capabilities. We are committed to solving problems in a way that exceeds expectations and delivers value-based capabilities for our customers.

Senior Consultant - Enterprise Data Systems

clear measures

I am a Principal Consultant with expertise in Data Warehouse building, Data Vault Methodology, Data Science, Business Analytics, Hadoop infrastructure, and Reporting.

I worked on multiple architecture projects, designing Data Vault Data Warehouse systems. Integrating Hadoop into an overall Logical Data Warehouse strategy for a major Retailer. Teaching internal sessions on Hadoop and Data Vault. I also built a Hadoop Data Science Lab for internal research and training.

Principal Consultant - Enterprise Architecture

3M Health Care

I engineered the cloud based infrastructure for the management of health care data for 3M customers. Using Rightscale to manage our clouds (Rackspace,Azure,AWS), and direct configuration tools for our hosted data centers. Our data platforms are diverse, DataStax Cassandra Enterprise, SQL Server,  Cloudera Hadoop, and Percona MySQL. I do data server configuration, capacity planning, data modeling, Pentaho Deployment, Performance tuning, support of our Data Science Lab and on-call support as the need arises. 
 
The typical duties of your average data-guy.

Lead Data Engineer

Rent The Runway

This was a research project to determine how Pentaho Business Intelligence platform could assist in the Data Science efforts of the analytics group. I deployed the platform, and we evaluated it based on comparison with the internally developed tools the analytics team had already built.

Data Management Consultant

Data isn’t just numbers — it is the new gold. 
As a passionate and deeply knowledgeable data scientist, I have vast expertise enhancing, automating, and analyzing data processes. Since I first ignited my data career in the US Marine Corps, governing global operational systems in support of various missions — an experience that embedded my foundational knowledge of data management, helped define my personal philosophy of ensuring data integrity, and shaped my professional principles and perspective of data protection — I have been solving data challenges for businesses of all sizes by diving deep and transforming data to elevate productivity, efficiency, growth, and success. 

Some notable achievements include:
•	Excelling as the principal problem-solver conveying analysis and ensuring effective platform function utilizing Snowflake, as well as educating customers on how to reduce their process run time by 92% –  from 1.5 hours down to just seven minutes – for DataOps.live. 
•	Stabilizing KORE Wireless’ server with configuration changes from day 1, resulting in zero outages and performance issues as opposed to consistent ecosystem outages averaging 2 hours per week across 600 databases. 
•	Catalyzing Markov chain model to identify high-performing students and engineering Ensemble Predictive Model using SQL rules and R’s decision tree to predict behaviors exhibited by at-risk student with greater than 80% accuracy for edtech giant Cengage Learning. 
•	Presenting at prestigious national data conferences and developing presentations on topics such as data structure graphs, high-level graph theory, and network analysis involving Gephi, Graphx, and Giraph.
•	Authoring and publishing The Enrichment Game: A Story About Making Data Powerful about winning in business by understanding in-depth insights derived from data to create platforms for surprising future successes.

I am always interested in expanding my professional network and eager to discover what the future has to offer. If you would like to start a conversation that moves us towards purposeful action with data, reach out to me at dougthedataguy@gmail.com or here on LinkedIn. 

Technical Toolkit: 
SNOWFLAKE Certified Professional
LINUX
KUBERNETES
ORACLE
R
MySQL
SQL SERVER
PYTHON
ClickHouse

Read more about how I make data powerful here: 
http://the-data-guy.blogspot.com/
https://github.com/dougneedham
https://technicspub.com/enrichment/
http://www.amazon.com/gp/product/B00XETTPCI

Doug Needham

On a whim, during my little vacation to Hawaii, I decided to hike Diamond Head. While it is not as tall as other mountains in the mainland, because it is a volcanic crater the only way to the top is to hike the side of the crater. This makes it quite steep. There were families, young couples, and even a few grandparents hiking the trail. Everyone goes at their own pace.

The trail is clearly marked, and while parts of it are rugged, there are handrails. Eventually there are metal stairs leading to concrete platforms at the top.

The view is spectacular!

As I huffed and puffed my way up the mountain, I had a realization. While many of us may be going the same direction, we are all doing it for our own reasons. Many times the trail is paved. Very seldom do we as engineers build something from scratch, brand new, that is so unique nothing like it has ever been seen before. 

We make incremental changes, sometimes the increment is small, sometimes it is a little bigger.

Most of my customers that I am advising are looking for these incremental improvements. They know a tweak here, an adjustment there, a little tuning over here will lead to ultimately big jumps in performance, reduced costs, increased stability, and better customer satisfaction.

It is exciting to see the peak of the mountain and know you want to get there. But the only way to get there is one step at a time, steady progress, and simply dont stop. Soon enough you will reach new heights, and be able to brag about your accomplishments.

If you are going in a direction I can help with, reach out. If you are not sure if I can guide you, reach out. Even if I can't help, I know many people who can.

Reach out to each other! 

I have friends, acquaintances, and colleagues here on this platform. 

Some are facing layoffs just before the holidays. Some are starting new businesses. Some have been out of work for some time. Some are working hard to improve their own lives and the lives of others. 

It would be nice to lend a helping hand to everyone we see who needs it.  We are not all in a position to help others. 

However, we can reach out and spend a little time chatting and supporting each other.  

I have been lucky enough to attend some networking events recently, and the conversations I have had with others have prompted me to follow up with some of those folks. Just hearing how things are going in their life, for good or ill, has been refreshing. 

Too often, I focus on my own situation, and my daughter's health issues can become all-consuming at times. I apologize to my circle for not saying hello directly for some time. I will work to do better. If we are connected, I hope you are doing well! If you are not doing well, I will do what I can for you, even if it is simply commenting on your posts so that more people see your struggles.  Feel free to message me if you'd like to chat. 

We can all do this for each other. We tend to focus on ourselves. Our product. Our offering. Our value. 

Yes, this is a professional site, and we all want to provide value and make money.  However, the occasional like, comment, repost, or message in support of our fellow humans never hurts. 

#MentalHealth #PeopleFirst.

Using AI to enhance the Design Thinking process to generate rapid prototypes that can be discussed and experimented with, was an engaging course from IDEO U. These courses are great for stretching your mental muscles before the end of the year.

People do not struggle with strategic decision-making; they make bad decisions, choose bad advisors, or miss opportunities right in front of their eyes.
 
Taking a leadership course through MasterClass  given by Navy SEALs, was very much a refresher for me. Marines and the SEALs have a very similar language given we have a common heritage of the Sea. The stories spoken about during the class brought back both memories both of my time in the Marines, as well of some experiences in business. 

I have some fond memories of seeing great leadership at work and being able to accomplish amazing things. 

I have plenty of not so fond memories where experiences of poor leadership caused issues that could have been avoided. 

I recommend this class, especially in the case where you are in leadership and are quite confident you do not need this.

#Leadership

End-of-year classes are supposed to be thought-provoking, or at least a new experience. 

The IDEO U courses on strategy using design thinking were certainly thought-provoking, and they showed me striking similarities with the Data Science process I have learned and worked with over the years. 

There is much hype, both good and bad, about what AI can do; there are still humans behind the curtain doing real work to ensure that AI systems are functioning and doing what they are supposed to. 

Design Thinking and proper Data Science processes start well before anyone's fingers touch a keyboard. 

It starts with: "Are we focused on the right strategy?" (probably not). "What are our customers saying?" "What are our customers ' or potential customers' needs, wants, pain points, or desires?"

Going the wrong direction faster is still the wrong direction. Taking slow, incremental steps going the right direction will probably get you to the success you are after. BTW, you did define your success criteria before you started, right? 

IDEO offers excellent classes and an interesting approach to solving these problems.

Is the end-of-year time frame one in which you take new classes, take time off, read new books, or try new projects? How do you wrap up the year?

A ringtone is twice as loud in the middle of the night.

Funny enough, the calls do not go to executives. Nor are project managers bothered at night. Developers have long since signed off for the day. Alerts, whether automated or manual, are escalated to those who can actually fix the problems. The problems must be fixed immediately. 

For the 911 of the enterprise, the data must flow.

When you build on a platform that has stood the test of time for even the demands of Snowflake, then you know when you sign off for the day. Your processes will run resolute, and only your personal alarm will awaken you. 

What do you bet your sleep on? 

#DataOps #DataOperations #Data #SleepBetter

❄️ Snowflake Data Superhero ❄️ Field CTO @ DataOps.live 🔹 Sr Data Professional 🔹 Data Modeling Aficionado 🔹 DV 2.0 Certified 🔹Datafluencer™️

🎁Give the Gift of Snowflake Automated CI/CD This Holiday Season 🎁

Snowflake practitioners this gift is for you! if you’re looking for the perfect gift to accelerate data development and deplyment into Snowflake this year, here it is: https://lnkd.in/e2WrDvWU

😇 Automated CI/CD for Snowflake
😇 Agentic AI to accelerate development
😇 Built-in governance, testing & deployment auditing
😇 Free to use — even on a Snowflake Trial Account

This season, give your Snowflake pipelines the care and nurturing they deserve.
 🚫 No more DIY YAML gymnastics.
 🚫 No more fragile GitHub Actions.
 🚫 No more “hope and pray” deployments.

With DataOps.live Dynamic Delivery, your teams can finally operationalize Snowflake solutions with:
♾️ Continuous testing they don’t have to write
♾️Deployment automation your auditors will love
♾️Environment consistency your engineers will thank you for
♾️Guardrails that keep everything reliable and business-safe
♾️Agentic AI that handles the busywork so humans can innovate

It’s the gift that keeps on giving — all year long.

And the best part? You can try even test it on a Snowflake Trial Account 

Make 2026 the year your Snowflake data products ship faster, safer, and with way less manual effort.

Happy Holidays! 🎄✨

#Snowflake #DataOps #DynamicDelivery #AIReadyData #DataEngineering #CICD #Automation #HolidaySeason #DataProducts #AgenticAI #DataSuperHeroe
Nick HalseyJohn F. MarcheseSteve HuberKeith CrystalJose Contreras  MBADoug NeedhamMark BartloVeton (Tony) HajdariJennifer KindleGuy Adams

Joe Reis 🤓

Chief Operating Officer

DeepLearning.AI

Instructor

ThinkQuick, LLC

Data engineering and architecture consulting.

Owner

Ternary Data

Ternary Data are veteran data engineers and “recovering data scientists” who have firsthand experience with the challenges of succeeding with data. We offer practical data engineering advice.

O'Reilly Media

Author

University of Utah

Adjunct Professor

Big Squid, Inc.

* Consulted with Big Squid clients and sales prospects about how to implement and leverage data science and machine learning within their organization.
* Built and implemented data science curriculum and training for Big Squid's sales and consulting team.
* Advised engineering and product team on data science best practices, infrastructure, and algorithms.
* Engineered data science and predictive analytics solutions for clients.
* Certified Domo technical consultant.

Director of Predictive Analytics

Teem

* Built Teem's (formerly Eventboard, now WeWork) analytics and data platform from the ground up. The insights and data help optimize and empower the workplaces of many smart companies including Uber, LinkedIn, Netflix, Dropbox, Airbnb, Palantir, Pinterest, Twitter, etc, etc, etc.
* Built and managed a team of nearly a dozen brilliant engineers and data scientists.
* Oversaw all data science and data engineering initiatives.

Lead engineer - Analytics and Data Science

JobDash

* Refactored front end of application from Django into Angular
* Enhanced company's analytics capabilities via internal BI 
* Core technologies used: Django, Python, Angular

Software Engineer

Pure Predictive

* Worked on Pure Predictive, a cutting edge machine learning application. Focused heavily on solving the fundamental problem in which data scientists spending 80%+ of their time on preprocessing and munging data, and only a fraction of their time on analysis. 

* Core development and invention of automated data preprocessing and feature selection tools to transform data into machine readable format. End result: reduced the time spent on data science pre-processing and data munging from 20+ hours to around 1 minute.

* Project lead of a team consisting of top notch data scientists, senior software engineers, and Math PhDs.

* Heavily involved in machine learning.

* Technologies used: Python, Java, Hadoop, AWS, Unix

Skullcandy Inc.

* Demand planning, forecasting, and global supply chain management.

* Worked with key Asian vendors.

* Integrated Astro's supply and demand components into Skullcandy.

* Implemented MRP, forecasting processes, and supply chain management processes for their gaming division.

* Supported global product launches into major Consumer Electronics retailers (Best Buy, Target, Gamestop, and key international partners). 

* Key to the success of the hottest gaming headset of 2012 - the Astro A50.

* Experience with SAP, Oracle Demantra.

Operations/Supply Chain

Best-Selling Author | Data Engineer and Architect | Recovering Data Scientist ™ | Global Keynote Speaker | Professor | Podcaster & Content Creator

Helping data leaders and teams build systems that scale, with clarity, real-world insight, and a healthy dose of skepticism toward hype.

If you’re navigating the challenges of modern data and AI engineering, you’re not alone. Whether it’s designing resilient pipelines, managing organizational chaos, or separating signal from noise in the AI era, the goal is simple: get data from source to value, reliably and efficiently.

That’s where I come in:
- Co-author of the bestselling Fundamentals of Data Engineering (O’Reilly), used by 100,000+ professionals worldwide
- Instructor of Coursera’s top-rated Data Engineering course
- Host of a popular newsletter, podcast, and video series focused on real talk in data and AI
- Advisor and educator to top data and AI leaders across startups, enterprises, and VC firms

Looking to make better decisions with your data stack, your team, or your roadmap? Shoot me a message and let's chat.

Looking to make better decisions with your data stack, your team, or your roadmap? Subscribe to my newsletter for regular updates: https://joereis.substack.com/

AI agents will operate at a scale that humans won't match, and will need new ways of working with data.

In this episode, Ciro Greco (CEO @ bauplan) and I chat about why AI agents need a new lakehouse paradigm, next-gen "Git for data", and much more.

Links in the comments.

Joe Reis

If you’re looking for the world‘s best virtual conference on data observability, reliability, and trust by design, look no further than Sifflet’s upcoming Signals Conference.

I’ll be speaking about how you can build trust by design into your architecture with data modeling. And there are lots of other awesome talks by amazing practitioners and leaders.

When: November 17 to 20
Where: Virtual (link in the comments)

See you there!

taplio

The Battle

Doug Needham

Joe Reis 🤓

Followers

Posts

Engagements

Likes

Performance ScoreDoug Needham's LinkedIn Performance

Performance ScoreJoe Reis 🤓's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Doug Needham

Joe Reis 🤓

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Doug Needham's LinkedIn Performance

Performance ScoreJoe Reis 🤓's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreDoug Needham's LinkedIn Performance