I build AI products from zero to one, and I teach others how to do the same.Over 12+ years, I've taken AI/ML products from proof-of-concept to launch at Amazon, Nike, T-Mobile, and Disney. The work I love most is early-stage: validating ideas, getting executive greenlight, and shipping MVPs that prove value before scaling. At Nike, that approach drove a $100M annual reduction in product returns.I also teach what I build. I co-created and lead the AI Product Management Certificate at the University of Washington, where experienced PMs ship real GenAI products complete with evaluation plans and GTM strategies. Previously, I trained 4 cohorts at Product School.What I bring: the ability to translate messy AI capabilities into clear product value, align technical and business stakeholders, and move fast without breaking trust. I write about AI PM craft at aipmguru.substack.com.I'm currently exploring opportunities to lead 0→1 AI product work at companies solving meaningful problems. I'm especially energized by AI in healthcare and education, though I'm open to any domain where the product can genuinely improve lives. 

Shaili Guru

This message made my morning ☀️

A reader just shared that they landed an AI Product Manager role, and mentioned my content helped along the way.

Here's the thing though: I'm glad the content helped, but they put in the work.

The learning. The interviewing. The career pivot. That's all them.

Moments like this remind me why I write. Not to have all the answers, but to make the path a little clearer for folks navigating the transition.

If you're in the middle of your own AI PM journey right now, keep going. It's working.

AI Product Leader | 0→1 Builder & Educator | Ex-Nike, Ex-Amazon, Ex-Disney

My kids got Harry Potter Guess Who for Christmas, and watching them play gave me an idea. What if I built a digital version with our 46 family members? They could learn their relatives the same way they're learning about Hogwarts. I opened Bolt and started building. 34 versions later, I had a working game. But the real lesson wasn't about AI coding tools.

It was seeing PM fundamentals play out in real-time, compressed into one afternoon. Clear requirements prevented wrong interpretations. Defining the data model upfront avoided rewrites to the architecture. Thinking through edge cases early meant fewer fixes later. AI doesn't ask clarifying questions or push back on ambiguity. It just builds exactly what you describe.

Vibe coding is a mirror for PM discipline. It validated what I've learned across teams: clear requirements, defined data models, explicit acceptance criteria. The 4 patterns that made this work 👇

If I were teaching someone about AI product lifecycles for the first time, I'd start with POCs.
Not because they're the first step in a framework diagram.

Because they're where most AI products quietly die.

It reminds me of my research days working with Zebrafish embryos. You don't scale up a staining protocol before you've proven it works on a handful of samples.

Same logic applies to AI.

But here's what most teams get wrong: they run AI POCs like traditional software spikes. Build it, demo it, ship it.

AI doesn't work that way.

Your POC might hit 85% accuracy. Looks great in a demo. But what happens to the 15% that fails? Who handles those cases? What's the cost of being wrong?

These are the questions traditional PMs aren't trained to ask.

In my latest post, I break down:
→ Why POCs are the critical gate (not just another phase)
→ Three types of AI products and how they change what you're validating
→ The question most teams forget to ask before calling a POC "successful."

Link in comments 👇

Building your personal brand is hard. On LinkedIn, off LinkedIn—doesn't matter. It's just hard.

I have been at it for about a year now, and honestly? Still figuring it out.

I'm part of PDMA Seattle's committee, and our next session is with Leslie on how to make LinkedIn work for your PM career.

📅 Jan 14, 12-1 PM PST 🔗 https://luma.com/wno7fylk

Our last session on PM hiring trends had a great turnout, and everyone walked away with something useful. Expecting this one to be just as good.

Come through if you're trying to figure out the brand thing too.

Product Development and Management Association Seattle team - Anna Bigornia Sako Eaton Harmony Crawford Prabina B. Matt Foster Sahit G.  Banu R.

#productmanagement #linkedin #personalbrand

Open article: Stand Out With Your Brand Out. How Product Managers can grow their career using LinkedIn. · Zoom · Luma by luma.com

Stand Out With Your Brand Out. How Product Managers can grow their career using LinkedIn. · Zoom · Luma

AI Product Builder @ NanoKernel | Generative AI, AI Agents, AIoT, Responsible AI, AI Product Management | Ex-Apple, Accenture, Cognizant, Verizon, AT&T | I help companies build standout Next-Gen AI Solutions

I help visionary companies build standout AI Products. 

I am seeking a Senior IC AI Product Manager role at established AI-focused companies. 

I am a transformative Product and IT Executive, blending innovation with leadership across every role I've embraced. My journey from the technical trenches to the strategic echelons of Product Leadership, Solutions Architecture, and Enterprise Data Strategy has been fueled by an insatiable drive for growth, an unwavering focus on customer success, and a passion for adapting and thriving in diverse organizational landscapes.

From Fortune 50 giants to agile startups, my path has been marked by over 8 groundbreaking product launches and more than 20 transformative project deliveries. At the heart of these achievements lies a deep-seated belief in the power of innovative thinking, strategic execution, and the magic of collaboration across domains and disciplines. 

My mantra? Value creation, operational excellence, and nurturing talent are the cornerstones of my transformative leadership. 

My commitment to thought leadership has led me to author several books on AI and Data, secure several patents, and contribute to the knowledge pool with articles and papers featured in renowned journals, blogs, and media channels. Dive into my insights on LinkedIn and Medium at https://harsha-srivatsa.medium.com

Education is an ongoing journey for me, enriched by formal degrees in Engineering, Math, Computer Science, and Management, alongside continuous learning from prestigious institutions like MIT, Stanford, Kellogg, and IDEOU. This foundation empowers me to lead at the intersection of technology and business, navigating the complexities of AI, IoT, Cloud Platforms, and beyond with ease and insight.

My commitment extends beyond the professional realm. As a High School Educator with Microsoft's TEALS Program, a Master Gardener certified by University of California, a mentor for immigrants through Upwardly Global, and a volunteer with the Coast Guard Auxiliary, I believe in giving back, sharing knowledge, and supporting communities. I am a believer in the transformative power of technology, education, and community service. Let’s connect, collaborate, and create impact together.

Email:  harsha.srivatsa at connectwithharsha.info | Website: https://www.connectwithharsha.me

Harsha Srivatsa

Whoa! I just realized that my network connection crossed 10,000 recently! 
I am elated as well as overwhelmed that the connections count crossed the five figure mark. 

There has been a spurt in connection requests both incoming and sent by me. Many times I have not had the chance to personalize connection requests (LinkedIn on Mobile just sends the request) so I thank those who just accepted a blank connection request. 

Shoutout to my latest connection Ridhima Ranka as I write. 

Thanks you Brian Kalis and Brian Kemler for accepting my recent connection requests. 

I know my network is large but I shall do my best to make our connection meaningful and value added. 

Please DM or reach out if you want to chat with me, I can share what I know and help in any way. I shall surely put my priority and focus on growing our interactions in a meaningful way. 

As I mentioned earlier, following my mentor Ha Nguyen advice, I took the resolution to be a Go-Giver in 2026. I intend sincerely to do that. 

May we all live in interesting times.

Senior AI Product Manager | Generative AI, AI Agents, AIoT, Responsible AI, AI Product Strategy | Ex-Apple, Accenture, Cognizant | I help companies build, debug and launch standout AI products

I just applied to OpenAI Grove 🌱 https://lnkd.in/g2-ED-E8

From what I understand, Grove isn’t a traditional accelerator. It’s a 5-week builder program designed for people at the very start of the company-building journey—bringing together a small, high-signal cohort (~15 people) with workshops, office hours, and mentorship from OpenAI’s technical leaders, plus time to co-build and pressure-test ideas. 

Why I applied: because the next wave of AI isn’t only an algorithm story. It’s a physics story.
Power. Water. Cooling. Land. Community resources.

As AI infrastructure scales, we’re already seeing grid constraints, water stress, and real policy/community pushback. And yet… we’re also moving toward an AI++ world where AI spreads into edge, robotics, and cyber-physical systems. We can’t unlock that future with “more GPUs” alone. We need responsible scaling.

So my application is about what I’m calling Resource-Responsible AI:
 a Resource Ledger + “GreenOps for AI” cockpit that helps AI teams measure, attribute, and reduce energy / carbon / water footprint across training, inference, and agentic workflows—turning sustainability from a slide into an operational KPI.

This idea came straight from my work in AI Micro Data Centers—where resource constraints aren’t abstract… they show up every single day.

Now comes the hopeful part 🙏
 I’m genuinely praying / manifesting that I get to be one of the lucky ~15.

#OpenAIGrove #OpenAI #ResourceResponsibleAI #SustainableAI #AIInfrastructure #AIDatacenters #GreenOps #MLOps #AIOps #AIAgents

Human-in-the-Loop (HITL) vs Human-on-the-Loop (HOTL) in Multi-Agent Systems

While I was thinking, researching and writing out the section on Human involvement in MAS in my recent book - The Illustrated Guide to Multi-Agent Systems, I had an epiphany on how to design Human + AI agent teams that actually work in production.

Most teams pick ONE human oversight model and apply it everywhere. Most agent demos treat “human oversight” like a feature you add later.

That's a mistake.

In production Multi-Agent Systems (MAS), that’s backwards.
Humans aren’t just approvers at the end of a workflow. They’re embedded in:
 ✅ objectives
 ✅ constraints & permissions
 ✅ evaluation criteria
 ✅ escalation paths
 ✅ accountability

There are actually two operating modes:
HITL (Human-In-The-Loop)
 ➡️ the system pauses. No consequential action proceeds without explicit approval.
HOTL (Human-On-The-Loop)
 ➡️ the system runs, but humans supervise with the power to intervene, override, or kill-switch.

Here’s the key: Production MAS needs both — applied selectively, not uniformly.

Here's the framework that actually works:
Map every action on two dimensions:
→ Risk (low to high) → Reversibility (can you undo it?)
Then apply the right model:
🟢 Low risk + Reversible (draft reports, queries) → HOTL. Let agents run. Monitor for anomalies.
🟡 High risk + Reversible (budget proposals, config changes) → HITL with fast approval cycles. Human validates, but keep it tight.
🔴 High risk + Irreversible (fund transfers, data deletion, contracts) → HITL with enhanced guardrails. Multi-party approval. Audit trails. No shortcuts.

The cost of getting it wrong:
→ All-HITL? Approval bottlenecks. Humans become the constraint. You've built a system slower than doing it by hand.
→ All-HOTL? One retry storm cascades across agents. Customer-facing workflows fail before anyone notices.
→ Misaligned incentives? Agents optimize for speed, skip validation steps, and technically "succeed" while violating business intent.

The production-grade approach:
* Classify actions by risk × reversibility
* Set confidence thresholds per workflow
* Layer guardrails: policy gates → confidence scoring → runtime monitoring → escalation routing
* Reserve HITL for the ~2% that's genuinely high-stakes
* Close the feedback loop: every override becomes training data

The mark of production-grade MAS isn't how autonomous it is.
It's how effectively it embeds human judgment at every level—while still moving at machine speed.

It’s finally here. 🚢
After two months of building, tinkering, and turning real lessons into reusable patterns. 

 📘 The Illustrated Guide to Multi-Agent Systems is now live on Amazon.

2025 was the year of AI Agents. The next unlock is multi-agent systems—planner/router/executor/verifier—built to survive tools, evaluation, and production reliability. This is a production-grade approach to AI agent engineering.

What makes this book different:
 ✅ Visual-first (see the system)
 ✅ Production-first (quality, safety, cost, ops)
 ✅ Artifact-driven (plans, evidence, verification)

🎁 Includes a FREE MAS Toolkit + learning artifacts.

If you’re building AI agent products or leading AI teams shipping AI agent systems, this is for you.

Links + reviewer form in the comments 👇

No book gets written in isolation.

While working on The Illustrated Guide to Multi-Agent Systems, I went deep into several books that shaped how I think about agent architectures, orchestration, and what it actually takes to build systems that work beyond demos.

A few that made a real difference:
Victor Dibia's Designing Multi-Agent Systems taught me to think in protocols and system guarantees, not just frameworks. 
Sinan Ozdemir's Building Agentic AI showed me why agents break and how to design around it. 
Thomas R. Caldwell's AI Engineering Bible grounded me in what production-ready actually means. 
Sadie St Lawrence's work on AI orchestration reframed how I think about the role behind the systems. 
And Pascal Bornet, Melissa Pamela, Anjanava Biswas, and Wrick Talukdar each added layers to my understanding of business impact, control loops, and organizational transformation.

My writing is a synthesis of existing thinking and experiential learning. 

These authors did the hard work of documenting their thinking, and I'm grateful I could stand on their shoulders while building my own visual playbook.

If you're serious about multi-agent systems, these books belong on your shelf. See attached document. 

Many thanks to these Authors: 
Pascal BORNET
Rakesh Gohel
Tom Davenport
David De Cremer
Melissa Pamela
Anjanava Biswas
Wrick Talukdar
Victor Dibia, PhD
Thomas R Caldwell
Sinan Ozdemir

Awesome AI Agent Engineering Books

The moment when an epiphany about AI Agent Observability happened, is when I put down some notes for this post. 

Most AI Observability approaches today tell you what happened.
However, AI agents don’t fail like microservices or servers.  AI agent observability must tell you what’s about to break.

AI Agents don’t just crash or timeout. They loop, over-retry, thrash tools, or quietly escalate work to humans while appearing “healthy.”

While there is great guidance from Arize AI / LangChain / Langfuse / Hugging Face, I feel that Observability for AI agents has to move beyond traces, cost, and latency—and focus on agent behavior.

I liked how IBM framed it: 
"Evaluate agent performance by collecting data about actions, decisions and resource usage. It helps answer critical questions, such as:
Is the agent providing accurate and helpful answers?
Is the agent using processing power efficiently?
Is the agent using appropriate tools to fulfill its goals?
What are the root causes of issues with an agent?
Is the agent complying with AI ethics and data protection mandates?

With these insights, organizations can troubleshoot and debug issues more effectively and improve the performance and reliability of AI agents. 
"
Hence I came up with a framework to monitor, measure and report on AI Agent Observability. 

Four operator questions. Eight metrics. One goal: trustable autonomy.

1️⃣ Is it working?
 • Task Success Rate
 • Verifier Pass Rate

2️⃣ Is it struggling?
 • Tool Success Rate
 • Retry Rate
 • Loop Rate

3️⃣ Is it efficient?
 • Time to Result (median, not average)
 • Cost Proxy (tokens, tool calls, retries)

4️⃣ Does it need help?
 • Escalation Rate

What’s different here isn’t the metrics themselves—it’s what they represent.
These aren’t infrastructure metrics.
 They’re behavioral signals:
• Retry & Loop Rates expose reasoning breakdowns
 • Verifier Pass Rate applies truth pressure
 • Escalation Rate defines the autonomy boundary
 • Time & Cost reveal economic viability, not just performance

This is observability as operability.
Not:
“Can I see what the agent did?”
But:
“Can I trust what it will do next—and recover when it doesn’t?”

That’s how AI agents I think can earn trust at scale.

I have written more about it in my upcoming book "The Illustrated Guide to Multi-Agent Systems". 

#AIAgents #AgentObservability #AgentOps #LLMOps #ReliabilityEngineering #EvalOps #AIArchitecture

LLM / Gen AI security is a topic that I am not familiar with although I follow the  awesome writing by my friend Rammohan Thirupasur on this topic. 

So when Dipali M., my fav editor at Packt sent me this book, I read this with a beginner mindset. 

One of the concepts in the book - Trust boundaries - inspired me to think about an equivalent aspect for Multi-Agent systems which I parlayed into my recently published book on Multi-Agent systems. 

AI-Native LLM security is a very important topic with very few credible resources. So this book from Ken Huang / Vaibhav(VB) Malik Ads Dawson
is a seminal work in the field. 

AI-Native LLM Security (Packt, 2025) — a book that lands at exactly the moment many of us are moving from “LLM demos” to “LLM systems in production.”

The big shift is this: LLM security isn’t just AppSec with a new wrapper. Once you add RAG, tools, agents, and enterprise data, the attack surface becomes language-shaped—prompt injection, system prompt leakage, vector store weaknesses, and “agentic” misuse don’t fit neatly into old threat models.

What I liked about this book is its practicality: it treats security as an end-to-end product discipline (architecture → trust boundaries → controls → monitoring → incident response), and it gives teams a shared language through frameworks like OWASP Top 10 for LLM Apps—so security, engineering, and product can align without hand-wavy debates.

The Security Question Has Changed. Is Your LLM Architecture Security Ready? Here's a question I've been sitting with lately: If your LLM made one wron

Open article: Review: AI-Native LLM Security by Vaibhav Malik, Ken Huang, and Ads Dawson by Harsha Srivatsa • 4 min read

Review: AI-Native LLM Security by Vaibhav Malik, Ken Huang, and Ads Dawson

Attending CES 2026 at Las Vegas for the first time. Its exciting, overwhelming, lots to take in.. But its an AI show like no other. Especially given my interest in Physical AI. 

CES 2026 (Day 1 Jan 5 2026) made one thing clear: AI is growing hands, wheels, and sensors.

We’re watching the shift from “AI that talks” → AI that moves, sees, touches, and acts in the real world.

Here are the Physical AI signals I’m tracking from Jan 5, 2026:
1) NVIDIA is pitching the “factory for robots + AVs.”
Rubin + new AV models (Alpamayo) feels like NVIDIA saying:
👉 “If you want autonomy at scale, you’ll train + simulate on our full stack.”
This is less about one chip… and more about owning the pipeline (compute → simulation → deployment).

2) Home robotics is jumping from gadgets → generalist helpers.
SwitchBot’s Onero H1 is a bold move: a wheeled humanoid doing real chores using an on-device VLA brain (Vision + Language + Action).
Translation: it can see the world, understand instructions, and do stuff—without depending on the cloud every second.

3) The AV sensing war is getting spicy.
Teradar’s terahertz sensor is trying to blend “lidar-like detail” with “radar-like all-weather.”
Arbe + NVIDIA are pushing ultra-HD radar as something closer to a primary perception system (not just a backup).
Big theme: “lidar-class perception” without lidar constraints.

4) Drones are turning into flying edge-AI computers.
Ambarella’s CV5 powering an 8K 360 drone concept is a reminder that high-res + onboard AI is becoming the baseline.
But… regulation and supply chain policy may decide who actually wins the market.

5) Samsung is framing the home as a “physical workflow.”
Not just “smart devices.” More like AI companions that sense → decide → act across the home.

My takeaway:
Physical AI is now a platform game.
The winners won’t ship devices. They’ll ship systems: hardware + models + sensor stack + ecosystem + safety.

For Physical AI Product Builders:
Build sensor-agnostic perception (avoid single-modality lock-in).
Go on-device first, cloud assist second (latency + privacy + safety).
Treat policy/compliance as a product requirement, not a footnote.

I wonder which becomes the “default brain” for Physical AI in 2026–2027: on-device VLA, cloud copilots, or hybrid?

#CES2026 #PhysicalAI #EmbodiedAI #Robotics #HumanoidRobots #AutonomousVehicles #EdgeAI #SensorFusion #ComputerVision #AIHardware #SmartHome #AgenticAI

The art of constructing Golden Data Sets (using the 6V framework) 🧪✨

While I was thinking and researching about Golden Sets for Agent Evaluations, the following questions came to my mind: 
- What should be the size of the Golden Data Set for effective evaluations of an Agent application? 
- What should be the variety of the representative entries in the GDS? 
- For an multi-agent system, should be have Evals - GDS for each agent and also for the MAS end to end? 

I didnt find too many authoritative source and definitive guidance. But I was able to find brilliant snippets that helped me a lot. 
* Stella Wenxing Liu substack blog Data Science x AI had an awesome article on how big an evaluation test should be. 
* Talking to an AI architect inspired me to come up with the GDS coverage - Happy Path (50%), Edge cases (20%), Tool failure cases (10%), safety and regressions (15%, 5%). 

Then I had the epiphany for applying the classic Big Data framework to GDS. 
Most teams treat a golden dataset like a spreadsheet they “should probably have.”

In AI Agent world, a golden set is something else:
* It’s your definition of “good”.
* Your release gate.
* Your anti-regression insurance.

The “big data” thinking - the 6V's - is a solid approach to building high quality GDS. 

1) Volume — Do we have enough test cases?
2) Variety — Do we cover different kinds of reality?
3) Velocity — How fast does it go stale?
4) Veracity — Can we trust labels + scoring?
5) Value — Does the dataset protect what matters?
6) Variability — Do we capture ambiguity + distribution shift?

The MAS twist: Golden sets are two-layer
If you’re building multi-agent systems, the 6Vs naturally imply two eval views:
✅ System-level E2E golden set (user journey → final outcome)
✅ Per-agent golden sets (tight contracts → faster debugging + safer iteration)

One last point: you don’t always need massive GDS. Often, a small pilot (~20) tells you the score variance, and you can estimate an effective test size—sometimes ~100 is plenty, not 5,000.

I have covered this in detail in my newly published book "The Illustrated Guide to Multi-Agent Systems" - link in comments. 

#EvalOps #LLMEvaluation #AgenticAI #MultiAgentSystem

Yet another awesome session from SuperInterviews with Kasey Superinterviews and Jeremy Barton. 

Hear Deborah Liu say there is no such thing as true failure in interviews. Every interview is a prep to the next interview.

I am convinced that simulation based learning and exploration should be an integral part of AI agent building. 

I used to learn multi-agent systems like this:
📖 read an article, blog, paper, course material
🧠 seem to get it and nod my understanding
🧩 imagine how I can implement it into my product
🚀 vibe build a MAS prototype
😅 get surprised when I make it work for real 

The uncomfortable truth: I could design and vibe build MAS architecture… but I didn’t have full confidence in my system design until I could see the system behave under real conditions.

So I changed how I learn.
Now I learn MAS architecture like a pilot learns flight:
In a simulator.

Not “animated diagrams.”
A decision lab: inputs → choices → consequences → artifacts.
I pick a simple multi-agent use case (any domain works: support, finance, research, ops) and simulate the build approach and design decisions. 

This is what’s helping me become a Super IC AI PM — because the simulator makes the invisible… visible:

Tradeoffs stop being theoretical
Topologies aren’t “better/worse.” They’re trade spaces.
You can literally watch coordination tax show up, and see where the quality gains stop being worth the latency/spend.

Reliability becomes a systems skill
Not prompt luck.
I inject failures (noisy retrieval, flaky tools, partial context, conflicting agents) and learn what actually stabilizes systems: guardrails, budgets, circuit breakers, fallbacks, escalation paths.

Production thinking becomes muscle memory
EvalOps, observability, and safe rollout aren’t just checklists — they’re behaviors over time.
Shadow → Canary → Full becomes something you practice, not just plan.

My litmus test is simple:
If a simulation helps me make a better build and decision decision and teaches me why, it’s real learning.

Here are three examples of simulation based artifacts that helped in better MAS system design: 

1. The EvalOps & Observability Dashboard is an interactive simulation that implements the complete evaluation lifecycle for multi-agent AI systems. Manage golden test sets, monitoring real-time metrics (latency, accuracy, cost, safety) and debug with distributed traces. 
https://lnkd.in/gvEE67RY

2. The Failure Mode Diagnostic Lab is an interactive simulation that shows the four critical failure modes in multi-agent AI systems: Loops, Tool Thrashing, Cost Explosions, and Security Breaches. Instead of just reading about what goes wrong, you can watch failures cascade through real-time metrics and audit logs—then apply fixes and see them work.
https://lnkd.in/g2j6SbjZ

3. The Cost Explorer is an interactive simulation that makes multi-agent system costs tangible. It visualizes the five root causes of cost explosions—context snowballing, unlimited fan-out, missing caches, memory hoarding, and wasted verifications. 
https://lnkd.in/gBnxx7qt

Simulation Artifacts for Multi-Agent Systems Design

The Evolution of Evaluations

The evolution of evaluations is happening in plain sight:  we’re moving from LLM-as-a-Judge → Agent-as-a-Judge.

This recent paper - A Survey on Agent-as-a-Judge [https://lnkd.in/dh9ThVRR] is an awesome recent survey publication and captures the trend thinking quite well. 

Many people have commented and written about the paper albeit in a static commentary way but not delving into the nuances. With Google Dynamic views in Gemini, I have found a way to really understand an AI paper and understand the principles and working outlined in a paper. Instead of passive reading (or even “paper → code”), I’m building interactive simulation-based learning artifacts: https://lnkd.in/d42i3hgp

In this simulation based learning artifact, I simulate the judge’s workflow (plan → evidence → verify → deliberate → report), stress-test edge cases and failure modes, run “what-if” scenarios to expose hidden assumptions
and walk away with a mechanistic understanding of how this would behave in production. I have used a code generation use case to illustrate the difference between Agent-as-a-Judge and LLM-as-a-Judge. 

Please do check out Google Gemini Dynamic View artifact link above. 

For agentic evaluation specifically, simulation stimulated my thinking about: 
 What tools? What verification steps? What memory? What arbitration? What’s the stopping rule?

LLM-as-a-Judge was a huge leap: scalable, “near-human-ish” scoring, and a practical proxy for preference/reward signals. 

But it’s starting to hit a ceiling as evaluands become multi-step, tool-using, and domain-specialized. 

Where LLM-as-a-Judge breaks down
A single-pass judge tends to struggle with:
* Biases baked into the judge (e.g., rewarding verbosity, style, or its own output patterns). 
* “Looks correct” vs “is correct”: it often can’t verify claims against the real world, so you get hallucinated correctness. 
* Cognitive overload when the rubric has many dimensions → coarse scores that miss nuance. 
* Over-focus on final output, while failing to verify intermediate actions in complex workflows. 

What Agent-as-a-Judge changes
Agent-as-a-Judge turns evaluation into an active process, not a static opinion:
* Plans + decomposes evaluation into subtasks
* Uses tools to ground & verify (search, code execution, environment checks)
* Leverages multi-agent collaboration to reduce single-model blind spots
* Uses memory to preserve intermediate evidence and produce more traceable judgments 

The paper even frames a maturity curve:
Procedural → Reactive → Self-Evolving judge agents (from fixed workflows, to adaptive routing/tool use, to agents that can refine rubrics and learn over time).

If you’re building EvalOps for agentic systems, I think this shift matters:
the Judge is no longer a prompt — it’s an Agent system.

Like many others, I too am on the bandwagon of AI agent / AIoT agents. 
I have delved into the nitty gritty of ~10 multi-agent implementation - 2 in Healthcare, 1 in AI Infra and the rest across different use cases mostly for productivity and product management. 
One of the MAS is currently deployed and running in a Hospital system. 

From these experiences, I’ve learned the hard way that with multi-agent systems, “it works” in a demo doesn’t mean it will survive production.

The biggest gap isn’t tooling. It’s intuition—how agents behave over time when tools fail, handoffs break, loops run, costs spike, or security boundaries get fuzzy.

So I’m sharing a systems-thinking approach I’m excited about: Simulation-First Development. Before we prompt and build, we simulate. We practice the tradeoffs. We inject failures safely. We generate artifacts that become design records.

In the article below, I break this down and outline a 22-simulation roadmap that mirrors the full multi-agent lifecycle—from mental models all the way to EvalOps, scaling, and enterprise constraints.

I have documented the complete set of 22 simulation based learning and exploration artifacts that are enumerated in the roadmap described in the article. 

If you would like to take a look at these simulation artifacts, comment SIM below and I shall send you the link.

A Systems Thinking Approach to Vibe Engineering Multi-Agent Systems 85% of AI agent pilots never reach production. Not because the technology failed. 

Open article: Simulation-Based Multi-Agent Development by Harsha Srivatsa • 4 min read

taplio

The Battle

Shaili Guru

Harsha Srivatsa

Followers

Posts

Engagements

Likes

Performance ScoreShaili Guru's LinkedIn Performance

Performance ScoreHarsha Srivatsa's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Shaili Guru

Harsha Srivatsa

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Shaili Guru's LinkedIn Performance

Performance ScoreHarsha Srivatsa's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreShaili Guru's LinkedIn Performance