TrendyTech

Everything I do at Trendytech is just driven by just 1 goal 
'Changing Lives of my students with Data Engineering & Gen-AI'

Founder of Trendytech

ValuWiki

I’m supporting ValuWiki on the Data Engineering side - building reliable pipelines that keep research data organized and accessible.
On top of that, I’m advising the team on Generative AI, especially RAG tuning, to make the product smarter and more trustworthy.
Excited to bring my data + AI experience into a space where precision and clarity matter the most.

Technical Advisor

VMware

• Led the design and implementation of VMC on AWS Analytics, enhancing cloud data management capabilities.  
• Developed scalable data engineering pipelines utilizing cloud services, improving data processing efficiency.  
• Established robust health monitoring and alerting systems, ensuring optimal performance and reliability.

Member of Technical Staff, VMware

Cisco

• Developed and designed robust Big Data systems to analyze extensive sales data, driving actionable insights for decision-making.  
• Spearheaded Anaplan data integration initiatives from India, enhancing data flow and operational efficiency.  
• Collaborated with cross-functional teams to ensure seamless integration and alignment with business objectives.

Big Data Developer, Cisco

Hello Data Enthusiasts,

I am Sumit Mittal (Founder @ Trendytech.in), offering In-depth Big Data on Cloud & GenAI Trainings.

An NIT Trichy alumnus, I have worked with Top companies like Cisco & VMware before starting Trendytech.

My passion towards transforming lives of students, made me able to train:
- 30,000+ IT Professionals
- Working at Top Product based Companies like Microsoft, Walmart, Google, VISA, Mastercard and more

My structured way of preparing, ensures to give you exposure equivalent to a 4+ years actually experienced Data Engineering professional.

I have empowered Individuals to Transition & Grow Exponentially, through my below most popular programs:
-  Ultimate Data Engineering Program
-  Gen-AI Masters Program

Entire curriculum is designed basis industry requirement, covering most important tech stack like:
- Pyspark, Databricks, Data Factory, AWS Cloud Services, Data Modeling, System Design & a lot more
- Agentic AI, RAG, LangChain, Prompt Engineering and more

Do Visit my Website https://trendytech.in/ for more details

Sumit Mittal

For a very limited time, my entire 50-Days SQL Superstar Program is completely free.

The earlier SQL playlist helped millions, but interviews today need more depth. 

So I am rebuilding everything from scratch to help you crack top product based companies.

I am also organising it in one clean portal with daily videos, notes, datasets, quizzes and certificates.

18 days are already live.

Enrolment Link: https://lnkd.in/giyJuJeG

Free access available for a limited time.

These 50 days can change your life!

#sql #dataengineering #databases

Founder @ TrendyTech.in | Data Engineering & GenAI Mentor | 400K+ Followers | Shaping the Next Generation of Data Engineers

5 Free Courses that you can't miss if you are an aspiring Data Engineer.

1. Python for Data Engineers - https://lnkd.in/gwsTQRgR

2. Complete SQL For Data Engineers - https://lnkd.in/gEj-vWa9

3. Data Engineering mock Interviews - https://lnkd.in/gRNgf2WM

4. Azure Data Engineer Project - https://lnkd.in/gUeKwATM

5. Git & Git Hub for Data Engineers - https://lnkd.in/g6GTZh2D

I hope this will help all the aspiring Data Engineers.

PS~ My new Data Engineering Batch is starting on coming Saturday. DM to know more!

#bigdata #databricks #apachespark

Switch from Service based to Product based company

Cracking 4 offers and getting desired hike

Transition from ETL to Data Engineering after 6 years

Joining one of Big 4 companies, Deloitte with 100%+ hike

This is a glimpse of what my students shared with me on their achievements last week.

So Proud to see all of them achieving their goals with their hard work, consistent efforts and never giving up attitude.

- M S Naga Chaithanya switched from Wipro and joined product-based company Amgen India as Manager-Data Engineering.

- Apurva Gawande cracked 4 offers and joined EPAM Systems as Data Engineer.

- Anirudh Mathur after spending 6 years in ETL, now joined EPAM as Data Engineer.

- Ranvijay Singh joined Deloitte as Consultant Data Engineer.

Just one message to every Data Enthusiast in the closing month of 2025;
If these and 1000s of my students can transition and take their career years ahead, you too definitely can.

Trust yourself, have faith in almighty and give it a try.
There might me some delay but Success is 100% bond to come one day.

Some success stories don’t just inspire, they remind us why courage matters.

Journey of my student, Sanietahera is one of them.

Working as a Hadoop Developer at TCS, she felt her growth slowing down. Many people accept that reality but she didn’t.

With uncertainty, fear and a strong desire to grow, she made a tough decision -
to rebuild her career in Data Engineering from the ground up.

She strengthened her fundamentals, mastered cloud, became confident in PySpark, optimisation techniques and Azure Databricks, and went on to deliver complex projects at Fractal.

Recently her efforts turned into yet another milestone.

She joined Coforge as Lead Data Engineer with an exceptional hike.

When she says - Sumit Sir’s Big Data course helped me get industry ready and become a solid Data Engineer;
This fills me with lot more satisfaction that my teaching is impacting lives and career of many.

Her story is a reminder that:
- Progress begins the moment you stop holding yourself back
- Skills can change destiny

Proud of your journey, Sanietahera. Wishing you even bigger wins ahead.

For those who are looking to make a career in Data Engineering. My new batch is starting on coming Saturday. DM to know more.

Most people use Spark every day but very few understand what actually happens behind the scenes.

Let me explain it in the simplest way possible.

Lets say you have a 20 node spark cluster

Each node is of size - 16 cpu cores / 64 gb RAM

Let's say each node has 3 executors,
with each executor of size - 5 cpu cores / 21 GB RAM

Now let's answer the 5 below questions.

=> 1. What's the total capacity of cluster?

We will have 20 * 3 = 60 executors
Total CPU capacity: 60 * 5 = 300 cpu Cores
Total Memory capacity: 60 * 21 = 1260 GB RAM

=> 2. How many parallel tasks can run on this cluster?

We have 300 CPU cores, we can run 300 parallel tasks on this cluster.

=> 3. Let's say you requested for 4 executors then how many parallel tasks can run?

so the capacity we got is 20 cpu cores
84 GB RAM
so a total of 20 parallel tasks can run.

=> 4. Let's say we read a csv file of 10.1 GB stored in datalake and have to do some filtering of data, how many tasks will run?

if we create a dataframe out of 10.1 GB file we will get 81 partitions in our dataframe. (will cover in my next post on how many partitions are created)

so we have 81 partitions each of size 128 mb, the last partition will be a bit smaller.

so our job will have 81 total tasks.
but we have 20 cpu cores

lets say each task takes around 10 second to process 128 mb data.
so first 20 tasks run in parallel,
once these 20 tasks are done the other 20 tasks are executed and so on...

so totally 5 cycles, if we think the most ideal scenario.

10 sec + 10 sec + 10 sec + 10 sec + 8 sec

first 4 cycles is to process 80 tasks all of 128 mb,
last 8 sec is to process just one task of around 100 mb, so it takes little lesser but 19 cpu cores were free during this time.

=> 5. is there a possibility of, out of memory error in the above scenario?

Each executor has 5 cpu cores and 21 gb ram.
This 21 gb RAM is divided in various parts -
300 mb reserved memory,
40% user memory to store user defined variables/data. example hashmap
60% spark memory - this is divided 50:50 between storage memory and execution memory.

so basically we are looking at execution memory and it will be around 28% roughly of the total memory allotted.

so consider around 6 GB of 21 GB memory is meant for execution memory.
per cpu core we have (6 GB / 5 cores) = 1.2 GB execution memory.

That means our task can roughly handle around 1.2 GB of data.
however, we are handling 128 mb so we are well under this range.

Once you start seeing Spark this way, everything becomes simpler.
You understand why jobs run slower, 
why skew happens, 
how parallelism works, 
and how memory gets used.

This is the foundation behind writing efficient Spark code.

I hope you liked this scenario based explanation :)

#bigdata #apachespark

One more student joining as Senior Data Engineer!

congratulations Madhumitha S, I am really happy for you.

A message to all who feel they are stuck,

Keep working hard with right guidance & you will be amazed to see the results.

New Milestone Unlocked – Joined Tech Mahindra as Senior Data Engineer with 100% hike

If you’re in IT field, one thing is clear — continuous upskilling is not optional anymore.

A few years back, I was working as a System Engineer with limited exposure.

But I took a bold decision to transition towards Data Engineering — and honestly, that changed everything for me.

From learning Spark, optimisation techniques, complex data pipelines, Databricks & AWS through Sumit Mittal  ’s Big Data Program;
I got the confidence and the skills to land my first big break in this exciting domain.

Were weekends spent studying? Yes.
Was it worth it in the end? Absolutely yes.

Today, after gaining multiple project experiences, I’m super thrilled to share that I’ve joined Tech Mahindra as a Senior Data Engineer with a great hike.

A big thank you to everyone who supported me throughout this journey—and a special thanks to Sumit Sir, whose teaching style kept me motivated every single day.

The growth journey continues and a long way to go further.

The enrolments for my Free 50-Days SQL Superstar Program will close soon.

Over 7000 candidates have enrolled for it.

If you’ve been planning to start, this is the best time.

All video sessions, notes, datasets, quizzes, and certificates are already organised on the learning portal.

Once the window closes, new registrations won’t be accepted.

Enrol here: https://lnkd.in/giyJuJeG

These 50 Days can change your life!

Do Like & Repost so that maximum people can benefit from this.

#sql #database #dataengineering

What things in Gen AI a Data Engineer should know!

LLM - Large Language Model (act's like a brain)

This LLM has 2 main stages
- Pre training
- Fine tuning
although there are more.. 

In Pre training Big Companies will take a big chunk of data (100's of TB) available on internet, blogs, wikipedia etc and get a gist of it in a few GB's. 

Lets consider 70 Billion parameter model, each parameter here will be a float of 2 bytes and will take 140 GB.

This will take 10's of days and 1000's of heavy machines, and 10's or 100's of million dollars to do this and we get a base/foundation model.

This base model cannot answer question, rather if you give it a question it might give more questions.

Now, we can leverage the foundation model and develop something.

As a Data Engineer, 

=> Prompt Engineering is the easiest thing to do, but when creating applications you need to write it in a programatic way. writing better prompts with more context can help us get better results.

=> Retrieval Augmented Generation (RAG) - As a Data Engineer you can index your company documents, here you would require to clean the data and index it in a vector database.

Later when a prompt is given, it will try to do a simarity search and will search in your vector database. whatever results come will be augmented to your prompt and it's like more context is passed as part of the prompt for better results.

So, setting up a vector DB, cleaning documents, and storing in vector DB can be directly in a Data Engineers portfolio.

=> Model Fine Tuning - This is where the world is moving towards. Instead of striving for bigger and bigger base model, it has been seen that a decent size base model with a good fine tuning done for specific tasks can perform better.

So, we will take a foundation mode, and we might have 10's or 100's of thousands of quality questions and answers (high quality training data) on which we fine tune the model.

There can be different fine tuned models specific to each task.

Do not try using a single LLM for everything.

A better way is to use specialised LLM's fine tuned differently to do different stuff's.

A model might be good at summarisation, other might be good at judging which summaries are better.

Also the LLM's can be given access to various tools like google search, libraries to plot a graph, calculator etc in order to achieve a task.

=> Now when you develop complex application, it might happen you use a LLM for summarisation, and the output of this LLM goes to other LLM for judging. 
So there can be a big pipeline, this can be developed using langchain.

So I feel
- A basic understanding of terminologies around Pre training
- Vector Database
- RAG
- Model Fine tuning to some extent
- Lang Chain
- Agentic
are the things which at the minimum a Data Engineer should know.

But there are many more things within this, which I will cover in extreme depth in my "Gen AI for Data Engineers Program"

I hope you learnt something new!

A few months back, I received a message from a working professional.

He wrote:

"I sit quietly during Gen AI discussions at work. Everyone talks about RAG, embeddings, transformers. I don’t understand anything. I feel I am falling behind."

He wasn’t new to Data Engineering.
He had more than a decade of experience.
Great at ETL, cloud, pipelines.

But when Gen AI conversations started in his team,
he felt invisible.

He joined my program.

Week 1:
He struggled with the terms. Even simple concepts like embeddings felt intimidating.

Week 7:
He built his first RAG application.
He could finally connect how chunking, indexing, and retrieval work.

Week 10:
He presented a prototype to his leadership.
Not perfect, but functional.

Week 14:
The same person who once sat silently in meetings,
now leads the internal Gen AI initiative inside his team.

His exact words:
"Earlier Gen AI looked complex. Now I realize it was just unfamiliar."

He didn’t magically gain confidence.

Confidence followed clarity.

Clarity followed action.

Today he is the go-to person at his workplace for Gen AI.

Not because he is the smartest.
Because he started early and stayed consistent.

If you’re sitting quiet in meetings today,
you could be leading them six months from now.

All you need is a starting point.

DM to know more about my Gen AI Program!

We are implementing a Gen AI based solution where we are planning to ease out the work of technical Support team.

currently students raise a zoho support ticket through the learning portal. Our support team answers it. 

This is a perfect use case where Gen AI can help to ease out the load of support team. Also the turnaround time can be shortened.

Let me tell you how we are approaching this.

we are working on creating a chatbot (initially it will be support team facing), 
Later once tested and enhanced this will be facing the user directly.

We are taking the previous tickets (can be a complete thread of Question, answer, followup question, answer....) using Zoho API

The first time we pick lets say all the tickets for last 3 months.

One ticket(better represented in json as its a complete thread) can be thought of as one chunk.

we are planning to use MongoDB Atlas service as it can store Json efficiently along with embeddings.

so all the documents from Zoho are now in MongoDB and provide us vector search capability.

When a user type in the query we are using a LLM to reframe the question properly and also divide a question into relevant sub questions.

then for each generated question we are doing a vector search and fetching top 2 chunks for each question above a certain threshold.

Now we are framing the right prompt for RAG.

The final answer would be well aligned with the previous answers given by our team.

Challenges Faced:-
1. Web Scraping Zoho Data which took many attempts to fetch the full thread
 
2. Data Cleaning from Text, We used several regular expressions but most would cause data loss. 

3. Initially we tried Managing a database (pinecone) for vector storage and text files. Before we finally streamlined into MongoDB Atlas for simplicity and scalability. we wanted to tickets in Json format as its a full thread.

A few more things -

we are trying to make it a configuration driven architecture. If we want to enable chat history there will be just one config flag which can be set.

Also, we will be managing the prompts in Langchain hub, so that these can be versioned and any changes should not involve code changes.

We are using a smaller model as a LLM, as its very cost effective. Also we do not most likely need a powerful LLM as our answer should be coming from the context only.

Further enhancements -

what if the context is not present, then right now we will say "Don't know"

But going forward if context is not there, we will bring a agentic behavior so that it can do a google search and try answering it.

Also we want to add incremental context.

So both RAG + Agentic Behaviour can make it even more effective.

These days there is a very huge push from the management to work on Gen AI POC's.

Now with this post you will have an idea on what you can try building within your organization.

I hope this post should help all the Gen AI aspirants.

PS~ My new Gen AI batch is starting on Saturday. DM to know more!

taplio