DataOps.live

In this role, I excel at mentoring and educating customers regarding Amazon Web Services (AWS) and Azure configuration and installation of agent-based runner, adjusting to regulatory and security rules and ecosystems in finance, marketing, retail, healthcare, and e-commerce industries. I also develop best practices and manage platform projects utilizing Snowflake and support principal architects, data architects, data modelers, data engineers, the VP of IT infrastructure, and the Chief Information Officer (CIO). 
CORE ACCOMPLISHMENTS
•	Guided customers on how to consolidate data from various systems for greater data analyst palatability and usage.
•	Redirected and centered technical customers to focus on utilizing data structures and integration for business success. 
•	Exercised exceptional patience and demonstrated superb knowledge while guiding platform-training customers through month-long “handholding” onboarding phase with daily questions until self-sufficiency was forged; followed up accordingly.

Senior Solutions Architect

Team San Filippo

Chief Technical Advisor

EVERYTHING BUT THE HOUSE (EBTH)

Here, I automated and optimized business processes by instituting data science linear programming technique. Additionally, I utilized Python and operations research to assist with the optimization process. 
CORE ACCOMPLISHMENTS
•	Transformed process to automatically estimate item valuation based on historical auctions and selection of items presented for typical 7-day auctions. 
•	Reduced workload of merchandizing team dramatically, from 3 members spending 20 hours each weekend to just 30 minutes by single team member, freeing up valuable time for team to focus on more specialized auctions as well as estate sales.

Senior Data Scientist

Firstlight Media

I defined the Data Strategy for bringing data together from multiple applications to a consolidated enrichment platform that produces data visualizations. I designed and implemented the tools and techniques for producing consumer and content-based recommendations for our media player using graph technologies.

Data Strategist

KORE Wireless

I modernized our SQL Server environment to apply best practices in database use. 

I act as a Data Science Mentor, and Data Architect as we design new applications and use-cases for our data products. 

I implemented consolidated management of our SQL Server infrastructure using Idera tools. Upgraded our environment to current versions of SQL Server. 

I built an ETL team for managing the integrations that tie our business systems Netsuite and Salesforce to our internally developed applications. 

Using Signal analysis, I was able to demonstrate capabilities that could be implemented for any kind of anomaly detection as we monitor devices for our customers.

Senior Manager Data Services.

Cengage Learning

I created an ensemble predictive model using SQL rules, and R decision tree's to accurately (Precision/Recall both above 80%) predict the interactions with our platforms that at risk students exhibit. I created a Semantic analysis process for survey data to quickly parse through hundreds of feedback comments to focus customers on the issues and sentiment most important to respondents.

Data Scientist

Illumination Works

I managed our Data Science lab. Our Data Science Lab using both Map Reduce and Spark is based on CDH 5 from Cloudera. It is built using Cloudera Director and Cloudera Manager in AWS. Presentations built using our Data Science Lab were presented in December 2014 and January 2015.

Illumination Works is a solution enabling company. We provide professional consulting, design, and development services that deliver core information and insight capabilities. We are committed to solving problems in a way that exceeds expectations and delivers value-based capabilities for our customers.

Senior Consultant - Enterprise Data Systems

clear measures

I am a Principal Consultant with expertise in Data Warehouse building, Data Vault Methodology, Data Science, Business Analytics, Hadoop infrastructure, and Reporting.

I worked on multiple architecture projects, designing Data Vault Data Warehouse systems. Integrating Hadoop into an overall Logical Data Warehouse strategy for a major Retailer. Teaching internal sessions on Hadoop and Data Vault. I also built a Hadoop Data Science Lab for internal research and training.

Principal Consultant - Enterprise Architecture

3M Health Care

I engineered the cloud based infrastructure for the management of health care data for 3M customers. Using Rightscale to manage our clouds (Rackspace,Azure,AWS), and direct configuration tools for our hosted data centers. Our data platforms are diverse, DataStax Cassandra Enterprise, SQL Server,  Cloudera Hadoop, and Percona MySQL. I do data server configuration, capacity planning, data modeling, Pentaho Deployment, Performance tuning, support of our Data Science Lab and on-call support as the need arises. 
 
The typical duties of your average data-guy.

Lead Data Engineer

Rent The Runway

This was a research project to determine how Pentaho Business Intelligence platform could assist in the Data Science efforts of the analytics group. I deployed the platform, and we evaluated it based on comparison with the internally developed tools the analytics team had already built.

Data Management Consultant

Data isn’t just numbers — it is the new gold. 
As a passionate and deeply knowledgeable data scientist, I have vast expertise enhancing, automating, and analyzing data processes. Since I first ignited my data career in the US Marine Corps, governing global operational systems in support of various missions — an experience that embedded my foundational knowledge of data management, helped define my personal philosophy of ensuring data integrity, and shaped my professional principles and perspective of data protection — I have been solving data challenges for businesses of all sizes by diving deep and transforming data to elevate productivity, efficiency, growth, and success. 

Some notable achievements include:
•	Excelling as the principal problem-solver conveying analysis and ensuring effective platform function utilizing Snowflake, as well as educating customers on how to reduce their process run time by 92% –  from 1.5 hours down to just seven minutes – for DataOps.live. 
•	Stabilizing KORE Wireless’ server with configuration changes from day 1, resulting in zero outages and performance issues as opposed to consistent ecosystem outages averaging 2 hours per week across 600 databases. 
•	Catalyzing Markov chain model to identify high-performing students and engineering Ensemble Predictive Model using SQL rules and R’s decision tree to predict behaviors exhibited by at-risk student with greater than 80% accuracy for edtech giant Cengage Learning. 
•	Presenting at prestigious national data conferences and developing presentations on topics such as data structure graphs, high-level graph theory, and network analysis involving Gephi, Graphx, and Giraph.
•	Authoring and publishing The Enrichment Game: A Story About Making Data Powerful about winning in business by understanding in-depth insights derived from data to create platforms for surprising future successes.

I am always interested in expanding my professional network and eager to discover what the future has to offer. If you would like to start a conversation that moves us towards purposeful action with data, reach out to me at dougthedataguy@gmail.com or here on LinkedIn. 

Technical Toolkit: 
SNOWFLAKE Certified Professional
LINUX
KUBERNETES
ORACLE
R
MySQL
SQL SERVER
PYTHON
ClickHouse

Read more about how I make data powerful here: 
http://the-data-guy.blogspot.com/
https://github.com/dougneedham
https://technicspub.com/enrichment/
http://www.amazon.com/gp/product/B00XETTPCI

Doug Needham

On a whim, during my little vacation to Hawaii, I decided to hike Diamond Head. While it is not as tall as other mountains in the mainland, because it is a volcanic crater the only way to the top is to hike the side of the crater. This makes it quite steep. There were families, young couples, and even a few grandparents hiking the trail. Everyone goes at their own pace.

The trail is clearly marked, and while parts of it are rugged, there are handrails. Eventually there are metal stairs leading to concrete platforms at the top.

The view is spectacular!

As I huffed and puffed my way up the mountain, I had a realization. While many of us may be going the same direction, we are all doing it for our own reasons. Many times the trail is paved. Very seldom do we as engineers build something from scratch, brand new, that is so unique nothing like it has ever been seen before. 

We make incremental changes, sometimes the increment is small, sometimes it is a little bigger.

Most of my customers that I am advising are looking for these incremental improvements. They know a tweak here, an adjustment there, a little tuning over here will lead to ultimately big jumps in performance, reduced costs, increased stability, and better customer satisfaction.

It is exciting to see the peak of the mountain and know you want to get there. But the only way to get there is one step at a time, steady progress, and simply dont stop. Soon enough you will reach new heights, and be able to brag about your accomplishments.

If you are going in a direction I can help with, reach out. If you are not sure if I can guide you, reach out. Even if I can't help, I know many people who can.

Reach out to each other! 

I have friends, acquaintances, and colleagues here on this platform. 

Some are facing layoffs just before the holidays. Some are starting new businesses. Some have been out of work for some time. Some are working hard to improve their own lives and the lives of others. 

It would be nice to lend a helping hand to everyone we see who needs it.  We are not all in a position to help others. 

However, we can reach out and spend a little time chatting and supporting each other.  

I have been lucky enough to attend some networking events recently, and the conversations I have had with others have prompted me to follow up with some of those folks. Just hearing how things are going in their life, for good or ill, has been refreshing. 

Too often, I focus on my own situation, and my daughter's health issues can become all-consuming at times. I apologize to my circle for not saying hello directly for some time. I will work to do better. If we are connected, I hope you are doing well! If you are not doing well, I will do what I can for you, even if it is simply commenting on your posts so that more people see your struggles.  Feel free to message me if you'd like to chat. 

We can all do this for each other. We tend to focus on ourselves. Our product. Our offering. Our value. 

Yes, this is a professional site, and we all want to provide value and make money.  However, the occasional like, comment, repost, or message in support of our fellow humans never hurts. 

#MentalHealth #PeopleFirst.

Using AI to enhance the Design Thinking process to generate rapid prototypes that can be discussed and experimented with, was an engaging course from IDEO U. These courses are great for stretching your mental muscles before the end of the year.

People do not struggle with strategic decision-making; they make bad decisions, choose bad advisors, or miss opportunities right in front of their eyes.
 
Taking a leadership course through MasterClass  given by Navy SEALs, was very much a refresher for me. Marines and the SEALs have a very similar language given we have a common heritage of the Sea. The stories spoken about during the class brought back both memories both of my time in the Marines, as well of some experiences in business. 

I have some fond memories of seeing great leadership at work and being able to accomplish amazing things. 

I have plenty of not so fond memories where experiences of poor leadership caused issues that could have been avoided. 

I recommend this class, especially in the case where you are in leadership and are quite confident you do not need this.

#Leadership

End-of-year classes are supposed to be thought-provoking, or at least a new experience. 

The IDEO U courses on strategy using design thinking were certainly thought-provoking, and they showed me striking similarities with the Data Science process I have learned and worked with over the years. 

There is much hype, both good and bad, about what AI can do; there are still humans behind the curtain doing real work to ensure that AI systems are functioning and doing what they are supposed to. 

Design Thinking and proper Data Science processes start well before anyone's fingers touch a keyboard. 

It starts with: "Are we focused on the right strategy?" (probably not). "What are our customers saying?" "What are our customers ' or potential customers' needs, wants, pain points, or desires?"

Going the wrong direction faster is still the wrong direction. Taking slow, incremental steps going the right direction will probably get you to the success you are after. BTW, you did define your success criteria before you started, right? 

IDEO offers excellent classes and an interesting approach to solving these problems.

Is the end-of-year time frame one in which you take new classes, take time off, read new books, or try new projects? How do you wrap up the year?

A ringtone is twice as loud in the middle of the night.

Funny enough, the calls do not go to executives. Nor are project managers bothered at night. Developers have long since signed off for the day. Alerts, whether automated or manual, are escalated to those who can actually fix the problems. The problems must be fixed immediately. 

For the 911 of the enterprise, the data must flow.

When you build on a platform that has stood the test of time for even the demands of Snowflake, then you know when you sign off for the day. Your processes will run resolute, and only your personal alarm will awaken you. 

What do you bet your sleep on? 

#DataOps #DataOperations #Data #SleepBetter

❄️ Snowflake Data Superhero ❄️ Field CTO @ DataOps.live 🔹 Sr Data Professional 🔹 Data Modeling Aficionado 🔹 DV 2.0 Certified 🔹Datafluencer™️

🎁Give the Gift of Snowflake Automated CI/CD This Holiday Season 🎁

Snowflake practitioners this gift is for you! if you’re looking for the perfect gift to accelerate data development and deplyment into Snowflake this year, here it is: https://lnkd.in/e2WrDvWU

😇 Automated CI/CD for Snowflake
😇 Agentic AI to accelerate development
😇 Built-in governance, testing & deployment auditing
😇 Free to use — even on a Snowflake Trial Account

This season, give your Snowflake pipelines the care and nurturing they deserve.
 🚫 No more DIY YAML gymnastics.
 🚫 No more fragile GitHub Actions.
 🚫 No more “hope and pray” deployments.

With DataOps.live Dynamic Delivery, your teams can finally operationalize Snowflake solutions with:
♾️ Continuous testing they don’t have to write
♾️Deployment automation your auditors will love
♾️Environment consistency your engineers will thank you for
♾️Guardrails that keep everything reliable and business-safe
♾️Agentic AI that handles the busywork so humans can innovate

It’s the gift that keeps on giving — all year long.

And the best part? You can try even test it on a Snowflake Trial Account 

Make 2026 the year your Snowflake data products ship faster, safer, and with way less manual effort.

Happy Holidays! 🎄✨

#Snowflake #DataOps #DynamicDelivery #AIReadyData #DataEngineering #CICD #Automation #HolidaySeason #DataProducts #AgenticAI #DataSuperHeroe
Nick HalseyJohn F. MarcheseSteve HuberKeith CrystalJose Contreras  MBADoug NeedhamMark BartloVeton (Tony) HajdariJennifer KindleGuy Adams

Queensland Department of Education

Enterprise Information Architect

Make IT Work

Founder

Department of Justice and Attorney-General QLD

Data management policy, strategy, and architecture aligned to a unique and complicated business context. Establishing elementary business and information architectures. 
Designed and built near real time judicial data hub.

Information and data architect

Metro South Health

Preparing local GP referrers and specialist healthcare service providers to adopt the new Smart Referrals as one of Queensland's first health services to embrace Smart Referrals at scale

Project Manager

Guiding the data architecture and business requirements for the state wide Referral Services Directory as a part of the Smart Referrals program

Data Architect

Tech Mahindra

Agitating for a radical transformation to Suncorp's data infrastructure

Enterprise Architect

Sunshine Coast Hospital and Health Service

I contributed to the success of the establishment and commissioning of the Sunshine Coast University Hospital (SCUH) as an enterprise architect. This included application portfolio design, EMR strategy, data and BI architecture, integration architecture and data/BI architecture.
I designed and assured a comprehensive workforce identity and access management (IDAM) solution that automated role based resource access and management for all workers, regardless of engagement mode
I also pioneered a data warehouse solution that provided realtime visibility of patient flows and emergency wait times, visible across the organisation (5 hospitals). This solution employed the Data Vault 2.0 system of business intelligence. This was a QLD and possibly world first solution due to it's data being exclusively sourced from realtime HL7 v2 messages. This POC required 2FTE for 3 weeks.
This approach is now being employed across many data warehouse style projects throughout Qld Health.

Enterprise Data Architect

Flight Centre

Customer MDM
EDW and BI Strategy
Data Governance

Information Architecture and BI Strategy

Health Services Information Agency

Enterprise Architect (Information)

Department of Climate Change

Lowering Australia's carbon emissions by establishing the integrated climate change regulator with enterprise architecture techniques.  Incorporating CPRS, Renewable Energy (ORER) and NGER.  This includes developing and establishing enterprise architecture practices and frameworks to implement the strategic ACCRA blueprint.

Business Aspect

Strategic advice to clients on Enterprise Architecture methods, patterns and frameworks, with a focus on Information architecture and information management, Data architecture, Business Process Management (BPM) , Business architecture, data management, data migration.

Information Architect

I will lead your organisation with practical strategies and architectures that rapidly extract value from your existing data assets.

My specialties: Data and everything to do with data. technology enabled transformations, technology strategies and architectures, information governance, data management, Identity management, solutions integration and interoperability, cloud, legacy replacement and data migrations, spatial information systems (GIS), database design and admin. Experience with most of the tools, notations, methodologies and frameworks. I've been focussing my career around data management for almost 30 years in almost every project role imaginable.


(views expressed here are mine alone)

taplio

The Battle

Doug Needham

Russell Searle

Followers

Posts

Engagements

Likes

Performance ScoreDoug Needham's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Doug Needham

Russell Searle

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Doug Needham's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreDoug Needham's LinkedIn Performance