Paul Iusztin

🤖 Senior Machine Learning Engineer • MLOps • Contractor ~ Helping startups engineer production-ready ML/AI systems.

I'm the author of the bestseller LLM Engineer's Handbook, lead instructor of the Agentic AI Engineering course, founding AI Engineer of a San Francisco start-up, and obsessed with making knowledge accessible through AI.

With over 10 years of experience in AI and software and 20 apps shipped, I teach AI Engineering as I wanted to at the beginning of my career. End-to-end. From idea to production. From data collection to deploying, monitoring and evaluation. With a focus on AI principles, software patterns and infrastructure systems that will thrive in a future dominated by AI coding tools.

My ultimate goal is to help other engineers escape PoC purgatory and 10x their AI Engineering skills.

.

If you want to become an AI Engineer pro, join Decoding AI for exclusive content on designing, building, and shipping AI software that works: 

🗞️ https://www.decodingai.com

See more:

🌐 Homepage: https://www.pauliusztin.ai
📚 LLM Engineer's Handbook: https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/
🤖 Agentic AI Engineering Course: https://academy.towardsai.net/courses/agent-engineering?ref=b3ab31

.

✉️ For collabs, contact me at: p.b.iusztin@gmail.com

Senior Machine Learning Engineer

🟡 Scalable ML batch infrastructure for continuous training & inference

The client, a generative AI leader, struggled with manual training and calling their ML models. As a tedious and unscalable process, they needed to automate their ML processes completely and integrate them with their current web platform.

➞ 𝐃𝐞𝐬𝐢𝐠𝐧𝐞𝐝 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a modular 𝐏𝐲𝐭𝐡𝐨𝐧 𝐩𝐚𝐜𝐤𝐚𝐠𝐞 that composes the ML model required steps into 3 main batch pipelines that can easily be automated: feature, training, and inference, 𝐞𝐥𝐢𝐦𝐢𝐧𝐚𝐭𝐢𝐧𝐠 a 𝐭𝐞𝐝𝐢𝐨𝐮𝐬 𝟐-𝐡𝐨𝐮𝐫 𝐦𝐚𝐧𝐮𝐚𝐥 𝐩𝐫𝐨𝐜𝐞𝐬𝐬/𝐯𝐢𝐝𝐞𝐨.
➞ 𝐃𝐞𝐩𝐥𝐨𝐲𝐞𝐝 a scalable & cost-effective 𝐛𝐚𝐭𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 and 𝐂𝐈/𝐂𝐃 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 on top of AWS ECS & SQS that scales in and out based on the number of jobs, 𝐬𝐚𝐯𝐢𝐧𝐠 𝐭𝐡𝐨𝐮𝐬𝐚𝐧𝐝𝐬 𝐨𝐟 𝐝𝐨𝐥𝐥𝐚𝐫𝐬/𝐦𝐨𝐧𝐭𝐡.
➞ 𝐒𝐚𝐯𝐞𝐝 𝐦𝐨𝐧𝐭𝐡𝐬 𝐨𝐟 𝐟𝐮𝐭𝐮𝐫𝐞 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 by using good software practices to make the Python module & batch cloud infrastructure flexible and composable.
➞ 𝐁𝐮𝐢𝐥𝐭 a 𝐯𝐨𝐢𝐜𝐞 𝐬𝐲𝐧𝐭𝐡𝐞𝐬𝐢𝐬 𝐏𝐨𝐂 in ~1 week required to 𝐜𝐥𝐨𝐬𝐞 the 𝐝𝐞𝐚𝐥 with one of the 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐜𝐥𝐢𝐞𝐧𝐭𝐬.

⚫️ Big data ingestion & archiving pipeline

The client wanted to expand its platform features by securely and safely ingesting, preprocessing, and storing TB of data into their platform.

➞ 𝐂𝐨𝐧𝐭𝐫𝐢𝐛𝐮𝐭𝐞𝐝 to the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of a 𝐝𝐚𝐭𝐚 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐧𝐠 >𝟓 𝐓𝐁 of data in 𝐨𝐧𝐞 𝐠𝐨 in a reliable, fast, and secure way.
➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a scalable 𝐝𝐚𝐭𝐚 𝐩𝐫𝐞𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports multiple media types (images, videos, audio, 3D objects) and creates small preview files that are cheap to transfer, store and visualize. The 𝐨𝐫𝐢𝐠𝐢𝐧𝐚𝐥 𝐟𝐢𝐥𝐞𝐬 are 𝐚𝐫𝐜𝐡𝐢𝐯𝐞𝐝 on an affordable storage solution 𝐫𝐞𝐝𝐮𝐜𝐢𝐧𝐠 𝐜𝐨𝐬𝐭𝐬 𝐛𝐲 𝐮𝐩 𝐭𝐨 𝟖𝟏%.

I'm currently designing a second brain AI agent.

Here's a high-level architecture (steal it and build your own):

The goal is to create an agent that can reason over everything I’ve ever written, watched, or saved while automatically picking up facts, events, and preferences about me.

The agent should then utilize that memory to assist me in writing, thinking, and creating.

At the foundation of everything happening is GraphRAG.

Then we have 3 steps:

𝟭/ 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 = 𝗺𝗲𝗺𝗼𝗿𝘆

All notes, documents, and links flow through a GraphRAG ingestion pipeline:
 
• Extract entities and relationships
• Summarize and embed content
• Store it as a knowledge graph

This gives me three retrieval modes: semantic search, metadata search, and graph traversal.

That’s how the agent can respond to queries like, “Write a section on building a modular MCP server.”

→ Find MCP-related notes
→ Traverse connected ideas 1–2 hops away.

We then orchestrate the entire pipeline using Prefect, which gives us durable workflows with scheduling, retries, monitoring, and failure recovery.

𝟮/ 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹

On top of that memory sits the agent.

It can:
 
• Query the knowledge graph
• Update long-term memory (episodic, semantic, procedural)
• Call external tools like web search or image generation
• Use a smaller fine-tuned “LLM twin” to match writing style

The agent automatically learns my preferences, facts, and events and writes them back into the knowledge graph.

So it never reasons over everything.
It retrieves only the relevant slice of memory.

This inference loop is also orchestrated with Prefect, this time to make agent calls durable: retries, monitoring, and scalable inference.

𝟯/ 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝘃𝗶𝗮 𝗠𝗖𝗣

The agent is exposed as an MCP server using FastMCP (by Prefect).

This lets me interact with it directly from tools like Claude or Cursor, refining ideas conversationally instead of relying on one-shot prompts.

𝗧𝗵𝗲 𝗰𝗼𝗿𝗲 𝗶𝗱𝗲𝗮 𝗶𝘀 𝘁𝗵𝗶𝘀:

Effective agents are built with structured memory + reliable orchestration.

P.S. Have you tried to build a Second Brain Agent yet?

Senior AI Engineer • Founder @ Decoding AI • Author @ LLM Engineer’s Handbook ~ I ship AI products and teach you about the process.

If you’re building AI agents in 2026,

You need a faster way to make architecture decisions...

95% of agents never hit production.

And it's because teams pick the wrong architecture from day one.

They use workflows when they need an agent.
Multi-agent when a simple pipeline would have worked.
Autonomy where determinism was enough.

So we created something to help with that...

Introducing, Building AI Agents Architecture Cheat Sheet.

It was created by Louis-François Bouchard from Towards AI, Inc. in collaboration with me for our Agentic AI Engineering course that's coming this month.

The idea was to help you make the best architecture decisions.

Specifically, here's what it helps you do:
 
• Decide Workflow vs Single Agent vs Multi-Agent based on real constraints
• Avoid unnecessary autonomy that explodes cost, latency, and debugging time
• Apply the Autonomy Test to know when you actually need an agent
• Design thin agents + heavy tools for reliability
• Add validation loops, observability, and human in the loop (HITL) by design
• Pick the minimum architecture that works before scaling complexity

It also includes:
 
• A practical kickoff decision process
• Architecture questions with concrete build outcomes
• Clear signals for when multi-agent systems actually make sense
• Production rules we use when shipping real agentic systems

This is exactly the kind of reference you want before you write any code.

If you want it, here's how to get it...
 
Link: https://lnkd.in/eb_V3ck7

P.S. Huge credit to Louis-François Bouchard and the Towards AI, Inc. team for putting this cheatsheet together.

I noticed something interesting after publishing LLM Engineers Handbook:

AI engineering is still widely misunderstood.

Many people expected the book to be a step-by-step guide to fine-tuning LLMs.

And this makes sense to an extent...

Fine-tuning is the most visible part of the AI stack.

But this is not what makes you an AI engineer.
It's just part of the stack.

In fact, it's a small commoditized part of the snack.

These days, frameworks like Unsloth AI, Axolotl, and TRL from Hugging Face make fine-tuning relatively straightforward.

The real engineering work is about:
 
• Designing data pipelines that reliably collect, clean, and transform data
• Building infrastructure that scales, recovers, and stays observable
• Orchestrating RAG pipelines, agents, and workflows
• Managing datasets, lineage, evaluations, and monitoring in production

In other words, data is the bottleneck.

If you fine-tune on a random dataset and run a cookbook script, you have learned the “hello world” of fine-tuning.

The challenge is knowing what data you need, how to get it, how to structure it, and how to keep it fresh as systems evolve.

So if you want to grow as an AI or LLM engineer, your focus must shift.

Stop asking, “How do I fine-tune this model?”

Start asking, “How do I build a system that feeds the model the right data, at the right time, reliably?”

This is where production systems succeed or fail

Model architecture tweaks and novel training tricks are important, but that work sits closer to research than engineering.

Both matter.
They are just different jobs.

AI engineering is about shipping systems that work, scale, and survive real users and real data.

And once you see that clearly, your learning priorities change completely.

GraphRAG is becoming mandatory for shipping real-world AI agents.

Here’s why:

Shipping reliable AI agents is mostly a data and memory problem.

But the majority of engineers still treat it like a prompting issue.

An agent fails, and the instinctive reaction is:

“Rewrite the prompt.”
“Add more instructions.”
“Increase the context window.”

Sometimes that helps.
Most of the time, it doesn’t.

Because modern agents rarely fail due to a lack of reasoning power...

They fail because they’re forced to reason over too much, too little, or the wrong context.

As systems grow, context fragments:
 
• Knowledge lives across docs, tickets, dashboards, and chat logs
• Ownership and dependencies exist only in people’s heads
• Old workarounds outlive their original intent
• Stale or irrelevant data quietly pollutes the context window

Longer context windows didn’t fix this...
They just made the problem easier to hide.

Performance degrades.
Contradictions creep in.
The agent repeats itself, hallucinates, or acts on outdated assumptions.

This has a name: context rot.

The real questions shouldn't be towards the prompts.

They should be targeted at the architecture:
 
• How do you store organizational knowledge long-term?
• How do you retrieve only what’s relevant right now?
• How do you prevent bad data from contaminating the context window?
• How do you connect memory, ownership, dependencies, and history?

This is where knowledge graphs and GraphRAG become essential.

Instead of dumping documents into a prompt, you model your domain: services, teams, incidents, decisions (and how they relate).

This is exactly what I'll walk through at AI Superstream: Context Engineering by O’Reilly.

Specifically, we'll cover:
 
• When vanilla RAG is not enough
• How knowledge graphs act as long-term memory for agents
• How GraphRAG keeps context windows small but meaningful
• How to design agent memory that scales in real systems

If you’re building agents meant to survive real users, real data, and real time, this layer is non-negotiable.

Prompting is the surface.
Context engineering is the foundation.

📆: 26th February 2026

🔗: https://lnkd.in/exndWkcS

P.S. Seats are limited.

P.S.S. Huge thank you to Angelina Yang for inviting me to this panel and to O'Reilly for making this event possible.

Most production incidents aren’t hard to fix.

But they are hard to understand.

By the time an alert fires, the problem usually isn’t new…

Someone has seen it before.
There’s a workaround.
There’s history.

But that context is scattered across Slack threads, old runbooks, dashboards, and people’s heads.

As systems grow, this becomes the real failure mode.

Not broken code.
Broken memory.

And this is exactly why GraphRAG matters in production systems.

Without graph-based memory, engineers lose time reconstructing what the organization already knows, instead of fixing the issue at hand.

The way out isn’t more alerts or better dashboards…

It’s designing agents that carry organizational memory.

In the latest episode of Decoding AI Magazine, I walk through how to design a Production Engineer agent that reacts to alerts by:
 
• Identifying affected services and teams
• Understanding how failures propagate through dependencies
• Surfacing the context engineers usually spend hours digging for

The key insight is memory.

We use GraphRAG to model services, teams, incidents, and documentation as a connected system the agent can traverse when it matters most.

If you’re building agentic systems for production environments, this is the mental model that holds up at scale.

The full breakdown will go live tomorrow at 1 PM CET.

Subscribe to Decoding AI Magazine to get it delivered to your inbox.

Link → https://lnkd.in/et8fmmBQ

I was a true believer in clean architecture.

I honestly thought it would help me structure my Python AI projects.

This resulted in countless hours wasted obsessively dividing my projects into four folders:
 
• Domain
• Application
• Infrastructure
• Interface

It felt right.
But it was a complete nightmare to maintain.

It wasn’t until I realized that these layers should be virtual concepts that I finally cracked it.

Because Python doesn’t force structure on you.
It gives you freedom.

And that freedom is exactly why AI projects fall apart.

If you don’t impose some structure, you get spaghetti code.
If you impose too much structure, you get ceremony and friction.

Most advice pushes you to one extreme or the other.

Either rigid Clean Architecture, copied from Java, or tool-specific templates that lock you into a framework.

Neither worked for me.

What finally worked was treating clean architecture as a set of boundaries.

The domain became “pure business logic.”
The application became orchestration.
Infrastructure became swappable details.
Interfaces became replaceable entry points.

But none of that needed to be enforced dogmatically in folders.

Once I made that shift, things got simpler.

AI agents became reusable.
Workflows became easier to change.
Switching models or storage stopped rippling through the codebase.

If your Python AI project feels messy and fragile, or clean but painful to work in...

You’re probably applying the right ideas in the wrong way.

Btw, I wrote down the exact mental model and project structure that finally worked for me while building AI agents and workflows.

Sharing it here in case it helps someone else…

Link: https://lnkd.in/eGsU7ww3

P.S. If clean architecture ever made your Python project worse before it made it better, you’re not alone :)

I created an AI agent to write a Substack post (and it went viral).

Nobody even noticed it was AI. The secret?

How we evaluated the agent.

Most people think creating accurate agents is about prompt magic, but that's not the case.

Your evaluation process can make or break everything.

So here's the exact framework we used:

𝟭/ 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲

You can’t design agents or evals in the abstract.

For us, the requirements were explicit:
 
• 10–20 minute technical articles
• Markdown output
• Clear structure and narrative flow
• Mermaid diagrams
• Grounded research

The agent wasn’t allowed to “be creative.”
It was either correct or incorrect.

𝟮/ 𝗧𝗿𝗮𝗻𝘀𝗹𝗮𝘁𝗲 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗺𝗲𝗻𝘁𝘀 𝗶𝗻𝘁𝗼 𝗯𝗶𝗻𝗮𝗿𝘆 𝗰𝗵𝗲𝗰𝗸𝘀

We didn’t use vague 1–5 scores.

We wrote yes/no metrics:
 
• Did the structure match expectations?
• Did it follow the storyline?
• Did it stay grounded in provided research?
• Were diagrams, code blocks, and formatting correct?

Binary metrics force clarity.
They also make finding failures easier and debuggable.

𝟯/ 𝗦𝗲𝗽𝗮𝗿𝗮𝘁𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 𝗳𝗿𝗼𝗺 𝗻𝗼𝗻-𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵

Some things can be checked directly.
Others can’t.

So we designed two layers:
 
• With ground truth: direct comparison to references
• Without ground truth: style, input coherence, topicality

Both matter.

One anchors the system.
The other preserves flexibility where it’s needed.

𝟰/ 𝗕𝘂𝗶𝗹𝗱 𝗮𝗻 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲

Each metric had:
 
• A custom prompt
• 20–30 few-shot examples
• A binary decision
• A written reason

Why?

It tells you why something failed, not just that it did.

𝟱. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝘁𝗵𝗲 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿

We labeled another 20–30 samples ourselves.

Human vs model agreement became the metric.

Once that loop was stable, iteration became engineering:
 
• Swap models
• Push changes
• Score outputs
• Debug with confidence

This framework scales to any agent system.

And it’s exactly the kind of thinking we go deep on in the Agentic AI Engineering course.

We’ll be going live in February 🚀

If you want to learn how to:
 
• Build agents with real evaluation layers
• Design LLM-as-judge systems correctly
• Move from “it works” to “it’s reliable”
 
Join the waitlist here → https://lnkd.in/dtxZuZpb
 
P.S. If you’re building agents without evals, you’re in for a bumpy ride.

In 4 days, I will be live on the DataTalksClub podcast, taking live QAs on AI Engineering, Skill Stack, Agents, LLMOps, and How to Ship AI Products.

Together with Alexey Grigorev we will chat about: 

• The principles behind production-grade systems
• How to build reliable agentic workflows​
• ​The LLMOps practices that keep models stable​
• ​​The patterns I've learned from building more than 20 AI applications

Plus answering all your amazing questions!

The podcast is live, virtual and FREE on 26 Jan.

Join here: https://luma.com/fx3wplwn

See you there ✌️

Thank you Alexey Grigorev for inviting me! So excited about this one.

FastMCP is the best way to build or hook to MCP servers nowadays. At agent-to-tools or even agent-to-agent communication (where an agent is exposed as a tool), MCP is the clear winner. That's why I'm super thrilled to see how quickly FastMCP is adapting to this change with their dynamic features included in their freshly baked 3.0 version.

DAY 1: FastMCP 3.0 Beta is live 🚀

We’re kicking off this week with a rebuild of the FastMCP foundation.

The foundation is rebuilt to move beyond simple tool scripts and into composable AI infrastructure.

𝙼𝚊𝚓𝚘𝚛 𝚄𝚙𝚍𝚊𝚝𝚎𝚜 👇
🔌 Provider Architecture: A unified system that replaces duplicated code across features for flexible, dynamic server building.

👤 Dynamic Scoping: Servers can now show different tools to different users from a single endpoint.

🧩 Composite Apps: Combine tools from data, platform, and product teams into one authorized interface without collisions.

🔍  Context Optimization: New features for search, curation, and progressive disclosure to keep agent context windows clean.

💡 Agent Skills: Direct instructions on how and when an agent should use specific tools.

𝙱𝚊𝚌𝚔𝚠𝚊𝚛𝚍 𝙲𝚘𝚖𝚙𝚊𝚝𝚒𝚋𝚒𝚕𝚒𝚝𝚢: The core Pythonic API remains the same. This rebuild adds the infrastructure required for dynamic environments without breaking your current v2 servers.

🚀 Install the beta: 𝐩𝐢𝐩 𝐢𝐧𝐬𝐭𝐚𝐥𝐥 𝐟𝐚𝐬𝐭𝐦𝐜𝐩==𝟑.𝟎.𝟎𝐛𝟏
💬 Join the Discord: https://lnkd.in/gVH4jZvE
📖 Docs: https://lnkd.in/dCH4cFmA
📝 Announcement Blog: https://lnkd.in/gWx24H_r
✨ What's New Blog: https://lnkd.in/ghQDjqQF

𝒪𝓃ℯ 𝓂ℴ𝓇ℯ 𝓉𝒽𝒾𝓃ℊ.

FastMCP is how you BUILD MCP servers.

Tomorrow we're showing you how you take them to a whole new level. 🌅

I make learning Data Science effortless for you | Data Science @ Bosch

Hi 👋🏻
I'm Banias, AI Engineer at Netze BW.

Applying new technologies and methods in Data Science & AI is my passion.

Through my experience and continual learning, I've gained valuable insights and tips that I love sharing with Data Scientists, Machine Learning Engineers and builders like you.

If you're looking for practical advice, helpful resources, and articles on the latest trends or best practices in the field, you're in the right place.

Let's connect and stay updated on the exciting world of Data Science together. Feel free to reach out, read my articles, and let's grow our knowledge and network. :)


Note: My views are my own and don‘t represent my employer.

Banias Baabe

How do you hit 98.7% accuracy with RAG on complex documents?

You don't use vector search.

Traditional RAG often fails because semantic similarity isn't true relevance. For tough domains, you need a system that reasons.

This is where 𝗣𝗮𝗴𝗲𝗜𝗻𝗱𝗲𝘅 shines.

It's a new, vectorless RAG framework that mimics how a human expert analyzes a document.

Instead of chunking and embedding, 𝗣𝗮𝗴𝗲𝗜𝗻𝗱𝗲𝘅 builds a "table-of-contents" tree and uses tree search to find the most relevant information.

Why is this better?
✅ 𝗛𝘂𝗺𝗮𝗻-𝗹𝗶𝗸𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹: It simulates how experts navigate complex information.
✅ 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗣𝗿𝗼𝗰𝗲𝘀𝘀: The reasoning is traceable and interpretable. No more "vibe retrieval."
✅ 𝗡𝗼 𝗔𝗿𝗯𝗶𝘁𝗿𝗮𝗿𝘆 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴: It respects the document's natural structure.
✅ 𝗡𝗼 𝗩𝗲𝗰𝘁𝗼𝗿 𝗗𝗕: The system relies purely on LLM reasoning and document hierarchy.

This approach achieved state-of-the-art results on the FinanceBench benchmark.

🔗 Check it out: github(.)com/VectifyAI/PageIndex


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

I teach AI Engineers the concepts and tools they need to stay ahead | AI Engineer @ Netze BW

Running a 70B LLM on a single 4GB GPU is possible with 𝗔𝗶𝗿𝗟𝗟𝗠.

𝗔𝗶𝗿𝗟𝗟𝗠 optimizes inference memory usage so you can run big models without quantization, distillation, or pruning.

It runs inference layer by layer, loading only the layer that is needed for the next compute step, then freeing that memory before moving on.​

To make that practical, it first preprocesses the original Hugging Face weights and re-shards them by transformer layer (instead of giant multi‑GB shards), so each load is focused and smaller.​

This matters because disk I/O becomes the bottleneck when you stream weights.

The main thing that still sits in GPU memory is the KV cache for attention, which is much smaller than full model weights for typical context lengths.​

𝗔𝗶𝗿𝗟𝗟𝗠 also adds prefetching so it can overlap loading the next chunk of weights with the current compute, which helps claw back some speed.

And if you want more throughput, it supports an optional “model compression” mode using block-wise quantization of weights to shrink what needs to be loaded from disk, with the project claiming up to ~3x speedup and minimal accuracy loss.

🔗 Link to repo: github(.)com/lyogavin/airllm


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

Stop destroying your documents with 512-token chunks.

Traditional RAG is breaking your tables, splitting your diagrams, and mixing up your model specifications.

I just read a great article from Alex Vesa about a working data modeling approach for complex technical documents.

The problem? Most RAG systems treat a 500-page service manual like a blog post. They chunk blindly. A table that spans three pages gets shredded into meaningless fragments.

Here's what actually works:

✅ Pages, not chunks
Use full pages as your retrieval unit. That's how humans read technical docs anyway.
✅ Three-page sliding windows
Process each page with its neighbors (N-1, N, N+1). Now your system knows when content continues across boundaries.
✅ Treat tables like they matter
Extract structure. Generate metadata. Flatten for search. Keep all three representations.
✅ Multi-vector indexing
Dense embeddings for semantic search. Sparse vectors for exact matches. ColBERT for precision. Combine them.
✅ Domain-specific metadata
Model applicability, component types, cross-page relationships. Not just title and date.

Link ➙ https://lnkd.in/dJ9aJKAB

Thinking about using 𝗽𝗴𝘃𝗲𝗰𝘁𝗼𝗿 for your next project?

You might want to think again.

Here's the reality of using it in production:
1️⃣ 𝗜𝗩𝗙𝗙𝗹𝗮𝘁 𝗶𝗻𝗱𝗲𝘅𝗲𝘀 𝗻𝗲𝗲𝗱 𝗰𝗼𝗻𝘀𝘁𝗮𝗻𝘁 𝗿𝗲𝗯𝘂𝗶𝗹𝗱𝘀. New vectors throw off your cluster distribution over time.
2️⃣ 𝗛𝗡𝗦𝗪 𝗶𝗻𝗱𝗲𝘅𝗲𝘀 𝗮𝗿𝗲 𝗺𝗲𝗺𝗼𝗿𝘆 𝗵𝗼𝗴𝘀. We're talking 10GB of memory for hours just to build an index for millions of vectors.
3️⃣ 𝗜𝗻𝘀𝗲𝗿𝘁𝗶𝗻𝗴 𝗻𝗲𝘄 𝘃𝗲𝗰𝘁𝗼𝗿𝘀 𝗶𝘀 𝗮 𝗽𝗮𝗶𝗻. With IVFFlat, you have to reindex preiodically. With HNSW, you're looking at graph traversals and memory allocation for every new vector.
4️⃣ 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗶𝘀𝗻'𝘁 𝗰𝗹𝗲𝗮𝗿-𝗰𝘂𝘁. Pre-filtering is great for selective filters, but post-filtering can give you incomplete results, and it only gets worse with more filters.
5️⃣ 𝗡𝗼 𝗯𝘂𝗶𝗹𝘁-𝗶𝗻 𝗵𝘆𝗯𝗿𝗶𝗱 𝘀𝗲𝗮𝗿𝗰𝗵. You'll have to figure out how to balance vector similarity with text relevance and normalize scores on your own.

A new extension, pgvectorscale, does fix some of these problems with more memory-efficient indexing and better filter performance. But it isn't supported out-of-the-box on Azure and AWS, so you'd have to manage your own Postgres instance.

𝗽𝗴𝘃𝗲𝗰𝘁𝗼𝗿 is a great tool, but it's not a silver bullet. Make sure you're ready for the operational overhead before you commit.

🔗 Link to article: https://lnkd.in/d6jSRy2y


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

Tired of blocking I/O messing up your async Python apps?

Try 𝗮𝗶𝗼𝗳𝗶𝗹𝗲𝘀 - the game-changer for handling files in asyncio without the headaches.

It's an Apache-licensed library that wraps standard file ops into async magic.

Why you should use it:
• Mirrors Python's built-in file API, so it's super familiar.
• Supports buffered text, binary files, and even tempfiles asynchronously.
• Delegates to a thread pool to keep your event loop flying.


🔗 Link to repo: github(dot)com/Tinche/aiofiles

---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

Vector databases are bloated.

What if you could get the same power with 97% less space?

Try 𝗟𝗘𝗔𝗡𝗡, the world's smallest vector index.

It transforms your laptop into a private and powerful RAG system.

You can finally RAG on everything.
-> Your entire file system
-> Millions of local emails
-> 💻 Your entire codebase
-> 💬 All your chat histories

Why is 𝗟𝗘𝗔𝗡𝗡 interesting?
✅ 𝗜𝘁'𝘀 𝗶𝗻𝗰𝗿𝗲𝗱𝗶𝗯𝗹𝘆 𝗹𝗶𝗴𝗵𝘁𝘄𝗲𝗶𝗴𝗵𝘁: Index 60 million documents in just 6GB instead of 201GB.
✅ 𝗜𝘁'𝘀 𝟭𝟬𝟬% 𝗽𝗿𝗶𝘃𝗮𝘁𝗲: Your data never leaves your laptop. No cloud costs, period.
✅ 𝗧𝗵𝗲𝗿𝗲 𝗶𝘀 𝗻𝗼 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝗹𝗼𝘀𝘀: Get the same search quality as heavyweight solutions.

🔗 Link to repo: github(dot)com/yichuan-w/LEANN


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

Do you want to identify which lines use the most amount of memory in your Python script?

Try 𝗺𝗲𝗺𝗼𝗿𝘆_𝗽𝗿𝗼𝗳𝗶𝗹𝗲𝗿.

𝗺𝗲𝗺𝗼𝗿𝘆_𝗽𝗿𝗼𝗳𝗶𝗹𝗲𝗿 is a Python module to make a line-by-line analysis of memory consumption for Python functions.

Below you can see how to use memory_profiler within your Python script:

• Decorate the function you want to profile with @𝗽𝗿𝗼𝗳𝗶𝗹𝗲.
• Run the script by passing the option -𝗺 𝗺𝗲𝗺𝗼𝗿𝘆_𝗽𝗿𝗼𝗳𝗶𝗹𝗲𝗿 to load the 𝗺𝗲𝗺𝗼𝗿𝘆_𝗽𝗿𝗼𝗳𝗶𝗹𝗲𝗿 module.

Link to project: pypi(dot)org/project/memory-profiler


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

What, if an API call in your program fails?

Because of, let’s say, an instable internet connection?

This is not so uncommon.

You usually should have some retry mechanism in your program.

With 𝐭𝐞𝐧𝐚𝐜𝐢𝐭𝐲 in Python, this isn’t a problem anymore.

𝐭𝐞𝐧𝐚𝐜𝐢𝐭𝐲 offers a retrying behavior with a decorator with powerful features like:

- Define Stop Conditions
- Define Wait Conditions
- Customize retrying on Exception
- Retry on Coroutines

One of my favorite libraries!

Check it out: github(dot)com/jd/tenacity


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

OpenAI's Responses API isn't what they're telling you it is.

They are talking about how it's "faster, more flexible, easier" than the old chat completions API.

But here's what's really going on.

OpenAI hides their reasoning traces.

Unlike Claude, DeepSeek, or Qwen, you can't see GPT-5's chain of thought.

This creates a massive problem for developers using the stateless chat completions API.

Without access to reasoning traces, you can't pass them back into the conversation.

Result? GPT-5 performs worse than it should in your applications.

The Responses API is the workaround.

✅ It's stateful (OpenAI stores conversation history on their servers)
✅ They maintain the hidden reasoning traces in the backend
✅ They plug it into conversations automatically
✅ They strip it out before sending responses back to you

You get full model capability without ever seeing the reasoning.

Here's the uncomfortable truth:

The old stateless API is actually simpler.

Prefix caching works in both. Parallel tool calling works in both.

There's nothing inherently better about stateful architecture here.

The Responses API only exists because OpenAI won't show you what their models are thinking.

Meanwhile, Anthropic keeps their reasoning traces completely transparent.

If you're building with AI APIs in 2026, this matters.

The question is whether you're comfortable with your provider hiding critical information from you.

Link to article: https://lnkd.in/drZjD9Xj


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

AI makes your team faster.

But it also puts your data at risk.

Files are moving to new tools constantly.

AI models are training on things they shouldn't see.

And old security tools just can't see what is happening.

They are blind to the context that actually matters.

That is why I'm watching Cyberhaven’s launch on February 3.

They are revealing a new platform that combines data security and protection into one tool.

It's built to handle how messy and fast data moves today.

Join the live event to see:
📍 A live demo of this new AI-native platform.
📍 How to spot the few dangerous actions hidden inside millions of normal ones.
📍 Real stories from leaders on why old rules don't work anymore.

Don't wait for a data leak to find the holes in your security.

Grab your spot here: https://lnkd.in/dC-aGjjM

Struggling with slow data processing for AI training?

Try 𝗹𝗶𝘁𝗱𝗮𝘁𝗮, the Python library that transforms and optimizes datasets at scale!

𝗹𝗶𝘁𝗱𝗮𝘁𝗮 offers:
🟢 Streaming large cloud datasets without local downloads
🟢 Accelerating training by 20x with optimized data
🟢 Parallelizing tasks across multiple machines effortlessly
🟢 Securely handling sensitive data with advanced encryption

Perfect for images, text, video, and more.

Check it out: github(dot)com/Lightning-AI/litdata


---
♻️ Found this useful? Share it with another builder.
➕ For daily practical AI and Python posts, follow Banias Baabe.

taplio

The Battle

Paul Iusztin

Banias Baabe

Followers

Posts

Engagements

Likes

Performance ScorePaul Iusztin's LinkedIn Performance

Performance ScoreBanias Baabe's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Paul Iusztin

Banias Baabe

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Paul Iusztin's LinkedIn Performance

Performance ScoreBanias Baabe's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScorePaul Iusztin's LinkedIn Performance