Jeff Frick

I am not telling you something to sell you something. I am sharing something to entertain, educate, and get you to stop and think for a minute. | Top Voice | Friday Five | Turn the Lens | Work 20XX

With over 30 years of experience in the media industry, I am a podcast host and a media entrepreneur who is passionate about the future of work, leadership, and innovation. I have founded and led Menlo Creek Media, a company that helps leaders, marketers, and communications professionals navigate and leverage an AI-driven asynchronous world, by offering executive training for podcast hosts and guests, content creation and media library curation and amplification programs, and community development programs.

I also host two podcasts, Work 20XX and Turn the Lens, where I facilitate the spread of best practices, insights, learning, and wisdom on topics that matter to the modern workforce and society. I aim to be a trusted source of information and inspiration for my listeners, by featuring smart and diverse guests, engaging in lively and thoughtful conversations, and sharing nuggets of knowledge. My goal is to help people get more impact with their communications and content investments and to empower them to thrive in a time of rapid change and uncertainty.

Founder & Principal

Please visit www.MenloCreek.com for a more comprehensive list of services and resources. 

Helping Leaders, Marketers, and Communications professionals navigate and leverage an AI driver asynchronous world. 

Host of 'Work 20XX'  with Jeff Frick show on the Future of Work 
Host of 'Turn the Lens' with Jeff Frick on Leadership and of other topics of interest.

We are able to take a model that is trained on very general purpose physical interaction data, and we train on more and more of it. And every single task that we are tracking continues to get better. - Pete Florence

To be honest, it doesn't even really necessarily matter too much what the x axis is. Other than like the x axis is something that we can continually do more of and the y axis is some measure of how good the robots are. And we see that we can just continue to pour in more and more effort on that x axis and the y axis continues to get better and better. - Pete Florence

And that is like a landmark type of moment - Pete Florence
----
Landmark moment.

I was blown away when I saw the slide earlier in Pete's keynote. Not enough parameters, the models crap out (a technical term), but push past 7 billion, and the magic happens. Improve on one, improve on all. Pete and the team at Generalist recently announced a 10 billion parameters plus foundational model.

So excited to share this conversation with Pete Florence, Co-Founder and CEO of Generalist. Generalist is another example of the latest round of startups around 'Embodied AI', AI that does things in the environment, among us. Pete has a deep research foundation, and we retraced some of the key moments in getting from the highly specialized world of industrial robots, to the 'every use case is a corner case' of interacting in the physical world outside a pen, to respond to any number of task demands and unforeseen environmental challenges standing between the present, and completing the assigned task. 

As Pete said, take the model and make it the robot brain.
Make it big enough, and the time you spend training on one thing, improves the performance on all things... 
Bang. Stops you in your tracks. 

Thrilled to co-release my conversation with Pete Florence on Turn the Lens, in collaboration with Humanoids Summit, brought to you by ALM Ventures, from Humanoids Summit 2025 Silicon Valley, December 2025.

Full Length Conversation
Pete Florence: Generalist, Scaling Laws, Train One Improve All | Turn the Lens with Jeff Frick Ep46 
https://lnkd.in/gta8S7Da

In collaboration with Humanoids Summit, brought to you by ALM Ventures, 
from Humanoids Summit 2025 Silicon Valley, December 2025
Thanks Modar Alaoui and Jesica Chavez 🤖🎙️ Chavez 

For more on Humanoids Summit, including 
May 2026 Summit in Tokyo visit 
https://lnkd.in/ga487tGq 

For more from Humanoids Summit SV 
https://lnkd.in/geeUa5Dp  

#embodiedai #physicalai #humanoidssummit #turnthelens #robotics #humanoids #Generalist

• Embodied • Scaling • Policy • Dexterity • Trades • 

First Friday Five of 2026. Embodied AI, when AI has arms and legs and can do things. When every case is a corner case, never before experience in training. (for instance, EVERY Waymo ride)

Thrilled to co-release the first five interviews we did at Humanoids Summit SV in December in collaboration with Humanoids Summit, ALM Ventures, Modar Alaoui and team.

And so much more to come. 
#Embodied #Scaling #Policy #Dexterity #Trades #FridayFive #Newsletter #TurnTheLens #Work20XX

It's been a few weeks, but we're back in the saddle. Closed out last year with ten interviews from at the Humanoids Summit SV in Mountain View, CA at 

Open article: • Embodied • Scaling • Policy • Dexterity • Trades • by Jeff Frick • 5 min read

• Embodied • Scaling • Policy • Dexterity • Trades •

All movement generates emotions, no matter what that movement is. Because we used to get eaten. - Dr. Catie Cuan

When you're moving around people, it's not enough to think about efficiency. You have to think about efficiency and impression. - Dr. Catie Cuan

Until now, everything that moves in our environment is a part of nature. So you have evolutionary encoding, you understand a tree operating under the movement of the wind. But we don't have any priors for autonomously moving machines. People will create all kinds of stories about what that movement means. - Dr. Catie Cuan
--
People love to fill in the blanks.

We can't help but see patterns everywhere we look, even if they don't exist.
And we add emotion, anthropomorphization, and other details to create the comfort of an explanatory story for the confusing world around us.

Smart roboticists are getting ahead of this, trying to "set the story" up front, so people don't fill in the blanks with who knows what.
Choreography in robotics, think user experience in classical computing.

Full Length Conversation
Dancing with Robots: Dr. Catie Cuan on Choreorobotics and the Future of Robotics | Humanoids Summit, hosted by Jeff Frick 
https://lnkd.in/gThRTDWg

For more on Humanoids Summit 
https://lnkd.in/ga487tGq

#robotics #humanoidssummit #choreography #humanrobotinteraction #embodiedAI #UX #adoption #movementmatters #emotion #movement #perception #people #robots #choreorobotics

AI Product Builder @ NanoKernel | Generative AI, AI Agents, AIoT, Responsible AI, AI Product Management | Ex-Apple, Accenture, Cognizant, Verizon, AT&T | I help companies build standout Next-Gen AI Solutions

I help visionary companies build standout AI Products. 

I am seeking a Senior IC AI Product Manager role at established AI-focused companies. 

I am a transformative Product and IT Executive, blending innovation with leadership across every role I've embraced. My journey from the technical trenches to the strategic echelons of Product Leadership, Solutions Architecture, and Enterprise Data Strategy has been fueled by an insatiable drive for growth, an unwavering focus on customer success, and a passion for adapting and thriving in diverse organizational landscapes.

From Fortune 50 giants to agile startups, my path has been marked by over 8 groundbreaking product launches and more than 20 transformative project deliveries. At the heart of these achievements lies a deep-seated belief in the power of innovative thinking, strategic execution, and the magic of collaboration across domains and disciplines. 

My mantra? Value creation, operational excellence, and nurturing talent are the cornerstones of my transformative leadership. 

My commitment to thought leadership has led me to author several books on AI and Data, secure several patents, and contribute to the knowledge pool with articles and papers featured in renowned journals, blogs, and media channels. Dive into my insights on LinkedIn and Medium at https://harsha-srivatsa.medium.com

Education is an ongoing journey for me, enriched by formal degrees in Engineering, Math, Computer Science, and Management, alongside continuous learning from prestigious institutions like MIT, Stanford, Kellogg, and IDEOU. This foundation empowers me to lead at the intersection of technology and business, navigating the complexities of AI, IoT, Cloud Platforms, and beyond with ease and insight.

My commitment extends beyond the professional realm. As a High School Educator with Microsoft's TEALS Program, a Master Gardener certified by University of California, a mentor for immigrants through Upwardly Global, and a volunteer with the Coast Guard Auxiliary, I believe in giving back, sharing knowledge, and supporting communities. I am a believer in the transformative power of technology, education, and community service. Let’s connect, collaborate, and create impact together.

Email:  harsha.srivatsa at connectwithharsha.info | Website: https://www.connectwithharsha.me

Harsha Srivatsa

Whoa! I just realized that my network connection crossed 10,000 recently! 
I am elated as well as overwhelmed that the connections count crossed the five figure mark. 

There has been a spurt in connection requests both incoming and sent by me. Many times I have not had the chance to personalize connection requests (LinkedIn on Mobile just sends the request) so I thank those who just accepted a blank connection request. 

Shoutout to my latest connection Ridhima Ranka as I write. 

Thanks you Brian Kalis and Brian Kemler for accepting my recent connection requests. 

I know my network is large but I shall do my best to make our connection meaningful and value added. 

Please DM or reach out if you want to chat with me, I can share what I know and help in any way. I shall surely put my priority and focus on growing our interactions in a meaningful way. 

As I mentioned earlier, following my mentor Ha Nguyen advice, I took the resolution to be a Go-Giver in 2026. I intend sincerely to do that. 

May we all live in interesting times.

Senior AI Product Manager | Generative AI, AI Agents, AIoT, Responsible AI, AI Product Strategy | Ex-Apple, Accenture, Cognizant | I help companies build, debug and launch standout AI products

I just applied to OpenAI Grove 🌱 https://lnkd.in/g2-ED-E8

From what I understand, Grove isn’t a traditional accelerator. It’s a 5-week builder program designed for people at the very start of the company-building journey—bringing together a small, high-signal cohort (~15 people) with workshops, office hours, and mentorship from OpenAI’s technical leaders, plus time to co-build and pressure-test ideas. 

Why I applied: because the next wave of AI isn’t only an algorithm story. It’s a physics story.
Power. Water. Cooling. Land. Community resources.

As AI infrastructure scales, we’re already seeing grid constraints, water stress, and real policy/community pushback. And yet… we’re also moving toward an AI++ world where AI spreads into edge, robotics, and cyber-physical systems. We can’t unlock that future with “more GPUs” alone. We need responsible scaling.

So my application is about what I’m calling Resource-Responsible AI:
 a Resource Ledger + “GreenOps for AI” cockpit that helps AI teams measure, attribute, and reduce energy / carbon / water footprint across training, inference, and agentic workflows—turning sustainability from a slide into an operational KPI.

This idea came straight from my work in AI Micro Data Centers—where resource constraints aren’t abstract… they show up every single day.

Now comes the hopeful part 🙏
 I’m genuinely praying / manifesting that I get to be one of the lucky ~15.

#OpenAIGrove #OpenAI #ResourceResponsibleAI #SustainableAI #AIInfrastructure #AIDatacenters #GreenOps #MLOps #AIOps #AIAgents

Human-in-the-Loop (HITL) vs Human-on-the-Loop (HOTL) in Multi-Agent Systems

While I was thinking, researching and writing out the section on Human involvement in MAS in my recent book - The Illustrated Guide to Multi-Agent Systems, I had an epiphany on how to design Human + AI agent teams that actually work in production.

Most teams pick ONE human oversight model and apply it everywhere. Most agent demos treat “human oversight” like a feature you add later.

That's a mistake.

In production Multi-Agent Systems (MAS), that’s backwards.
Humans aren’t just approvers at the end of a workflow. They’re embedded in:
 ✅ objectives
 ✅ constraints & permissions
 ✅ evaluation criteria
 ✅ escalation paths
 ✅ accountability

There are actually two operating modes:
HITL (Human-In-The-Loop)
 ➡️ the system pauses. No consequential action proceeds without explicit approval.
HOTL (Human-On-The-Loop)
 ➡️ the system runs, but humans supervise with the power to intervene, override, or kill-switch.

Here’s the key: Production MAS needs both — applied selectively, not uniformly.

Here's the framework that actually works:
Map every action on two dimensions:
→ Risk (low to high) → Reversibility (can you undo it?)
Then apply the right model:
🟢 Low risk + Reversible (draft reports, queries) → HOTL. Let agents run. Monitor for anomalies.
🟡 High risk + Reversible (budget proposals, config changes) → HITL with fast approval cycles. Human validates, but keep it tight.
🔴 High risk + Irreversible (fund transfers, data deletion, contracts) → HITL with enhanced guardrails. Multi-party approval. Audit trails. No shortcuts.

The cost of getting it wrong:
→ All-HITL? Approval bottlenecks. Humans become the constraint. You've built a system slower than doing it by hand.
→ All-HOTL? One retry storm cascades across agents. Customer-facing workflows fail before anyone notices.
→ Misaligned incentives? Agents optimize for speed, skip validation steps, and technically "succeed" while violating business intent.

The production-grade approach:
* Classify actions by risk × reversibility
* Set confidence thresholds per workflow
* Layer guardrails: policy gates → confidence scoring → runtime monitoring → escalation routing
* Reserve HITL for the ~2% that's genuinely high-stakes
* Close the feedback loop: every override becomes training data

The mark of production-grade MAS isn't how autonomous it is.
It's how effectively it embeds human judgment at every level—while still moving at machine speed.

It’s finally here. 🚢
After two months of building, tinkering, and turning real lessons into reusable patterns. 

 📘 The Illustrated Guide to Multi-Agent Systems is now live on Amazon.

2025 was the year of AI Agents. The next unlock is multi-agent systems—planner/router/executor/verifier—built to survive tools, evaluation, and production reliability. This is a production-grade approach to AI agent engineering.

What makes this book different:
 ✅ Visual-first (see the system)
 ✅ Production-first (quality, safety, cost, ops)
 ✅ Artifact-driven (plans, evidence, verification)

🎁 Includes a FREE MAS Toolkit + learning artifacts.

If you’re building AI agent products or leading AI teams shipping AI agent systems, this is for you.

Links + reviewer form in the comments 👇

No book gets written in isolation.

While working on The Illustrated Guide to Multi-Agent Systems, I went deep into several books that shaped how I think about agent architectures, orchestration, and what it actually takes to build systems that work beyond demos.

A few that made a real difference:
Victor Dibia's Designing Multi-Agent Systems taught me to think in protocols and system guarantees, not just frameworks. 
Sinan Ozdemir's Building Agentic AI showed me why agents break and how to design around it. 
Thomas R. Caldwell's AI Engineering Bible grounded me in what production-ready actually means. 
Sadie St Lawrence's work on AI orchestration reframed how I think about the role behind the systems. 
And Pascal Bornet, Melissa Pamela, Anjanava Biswas, and Wrick Talukdar each added layers to my understanding of business impact, control loops, and organizational transformation.

My writing is a synthesis of existing thinking and experiential learning. 

These authors did the hard work of documenting their thinking, and I'm grateful I could stand on their shoulders while building my own visual playbook.

If you're serious about multi-agent systems, these books belong on your shelf. See attached document. 

Many thanks to these Authors: 
Pascal BORNET
Rakesh Gohel
Tom Davenport
David De Cremer
Melissa Pamela
Anjanava Biswas
Wrick Talukdar
Victor Dibia, PhD
Thomas R Caldwell
Sinan Ozdemir

Awesome AI Agent Engineering Books

The moment when an epiphany about AI Agent Observability happened, is when I put down some notes for this post. 

Most AI Observability approaches today tell you what happened.
However, AI agents don’t fail like microservices or servers.  AI agent observability must tell you what’s about to break.

AI Agents don’t just crash or timeout. They loop, over-retry, thrash tools, or quietly escalate work to humans while appearing “healthy.”

While there is great guidance from Arize AI / LangChain / Langfuse / Hugging Face, I feel that Observability for AI agents has to move beyond traces, cost, and latency—and focus on agent behavior.

I liked how IBM framed it: 
"Evaluate agent performance by collecting data about actions, decisions and resource usage. It helps answer critical questions, such as:
Is the agent providing accurate and helpful answers?
Is the agent using processing power efficiently?
Is the agent using appropriate tools to fulfill its goals?
What are the root causes of issues with an agent?
Is the agent complying with AI ethics and data protection mandates?

With these insights, organizations can troubleshoot and debug issues more effectively and improve the performance and reliability of AI agents. 
"
Hence I came up with a framework to monitor, measure and report on AI Agent Observability. 

Four operator questions. Eight metrics. One goal: trustable autonomy.

1️⃣ Is it working?
 • Task Success Rate
 • Verifier Pass Rate

2️⃣ Is it struggling?
 • Tool Success Rate
 • Retry Rate
 • Loop Rate

3️⃣ Is it efficient?
 • Time to Result (median, not average)
 • Cost Proxy (tokens, tool calls, retries)

4️⃣ Does it need help?
 • Escalation Rate

What’s different here isn’t the metrics themselves—it’s what they represent.
These aren’t infrastructure metrics.
 They’re behavioral signals:
• Retry & Loop Rates expose reasoning breakdowns
 • Verifier Pass Rate applies truth pressure
 • Escalation Rate defines the autonomy boundary
 • Time & Cost reveal economic viability, not just performance

This is observability as operability.
Not:
“Can I see what the agent did?”
But:
“Can I trust what it will do next—and recover when it doesn’t?”

That’s how AI agents I think can earn trust at scale.

I have written more about it in my upcoming book "The Illustrated Guide to Multi-Agent Systems". 

#AIAgents #AgentObservability #AgentOps #LLMOps #ReliabilityEngineering #EvalOps #AIArchitecture

LLM / Gen AI security is a topic that I am not familiar with although I follow the  awesome writing by my friend Rammohan Thirupasur on this topic. 

So when Dipali M., my fav editor at Packt sent me this book, I read this with a beginner mindset. 

One of the concepts in the book - Trust boundaries - inspired me to think about an equivalent aspect for Multi-Agent systems which I parlayed into my recently published book on Multi-Agent systems. 

AI-Native LLM security is a very important topic with very few credible resources. So this book from Ken Huang / Vaibhav(VB) Malik Ads Dawson
is a seminal work in the field. 

AI-Native LLM Security (Packt, 2025) — a book that lands at exactly the moment many of us are moving from “LLM demos” to “LLM systems in production.”

The big shift is this: LLM security isn’t just AppSec with a new wrapper. Once you add RAG, tools, agents, and enterprise data, the attack surface becomes language-shaped—prompt injection, system prompt leakage, vector store weaknesses, and “agentic” misuse don’t fit neatly into old threat models.

What I liked about this book is its practicality: it treats security as an end-to-end product discipline (architecture → trust boundaries → controls → monitoring → incident response), and it gives teams a shared language through frameworks like OWASP Top 10 for LLM Apps—so security, engineering, and product can align without hand-wavy debates.

The Security Question Has Changed. Is Your LLM Architecture Security Ready? Here's a question I've been sitting with lately: If your LLM made one wron

Open article: Review: AI-Native LLM Security by Vaibhav Malik, Ken Huang, and Ads Dawson by Harsha Srivatsa • 4 min read

Review: AI-Native LLM Security by Vaibhav Malik, Ken Huang, and Ads Dawson

Attending CES 2026 at Las Vegas for the first time. Its exciting, overwhelming, lots to take in.. But its an AI show like no other. Especially given my interest in Physical AI. 

CES 2026 (Day 1 Jan 5 2026) made one thing clear: AI is growing hands, wheels, and sensors.

We’re watching the shift from “AI that talks” → AI that moves, sees, touches, and acts in the real world.

Here are the Physical AI signals I’m tracking from Jan 5, 2026:
1) NVIDIA is pitching the “factory for robots + AVs.”
Rubin + new AV models (Alpamayo) feels like NVIDIA saying:
👉 “If you want autonomy at scale, you’ll train + simulate on our full stack.”
This is less about one chip… and more about owning the pipeline (compute → simulation → deployment).

2) Home robotics is jumping from gadgets → generalist helpers.
SwitchBot’s Onero H1 is a bold move: a wheeled humanoid doing real chores using an on-device VLA brain (Vision + Language + Action).
Translation: it can see the world, understand instructions, and do stuff—without depending on the cloud every second.

3) The AV sensing war is getting spicy.
Teradar’s terahertz sensor is trying to blend “lidar-like detail” with “radar-like all-weather.”
Arbe + NVIDIA are pushing ultra-HD radar as something closer to a primary perception system (not just a backup).
Big theme: “lidar-class perception” without lidar constraints.

4) Drones are turning into flying edge-AI computers.
Ambarella’s CV5 powering an 8K 360 drone concept is a reminder that high-res + onboard AI is becoming the baseline.
But… regulation and supply chain policy may decide who actually wins the market.

5) Samsung is framing the home as a “physical workflow.”
Not just “smart devices.” More like AI companions that sense → decide → act across the home.

My takeaway:
Physical AI is now a platform game.
The winners won’t ship devices. They’ll ship systems: hardware + models + sensor stack + ecosystem + safety.

For Physical AI Product Builders:
Build sensor-agnostic perception (avoid single-modality lock-in).
Go on-device first, cloud assist second (latency + privacy + safety).
Treat policy/compliance as a product requirement, not a footnote.

I wonder which becomes the “default brain” for Physical AI in 2026–2027: on-device VLA, cloud copilots, or hybrid?

#CES2026 #PhysicalAI #EmbodiedAI #Robotics #HumanoidRobots #AutonomousVehicles #EdgeAI #SensorFusion #ComputerVision #AIHardware #SmartHome #AgenticAI

The art of constructing Golden Data Sets (using the 6V framework) 🧪✨

While I was thinking and researching about Golden Sets for Agent Evaluations, the following questions came to my mind: 
- What should be the size of the Golden Data Set for effective evaluations of an Agent application? 
- What should be the variety of the representative entries in the GDS? 
- For an multi-agent system, should be have Evals - GDS for each agent and also for the MAS end to end? 

I didnt find too many authoritative source and definitive guidance. But I was able to find brilliant snippets that helped me a lot. 
* Stella Wenxing Liu substack blog Data Science x AI had an awesome article on how big an evaluation test should be. 
* Talking to an AI architect inspired me to come up with the GDS coverage - Happy Path (50%), Edge cases (20%), Tool failure cases (10%), safety and regressions (15%, 5%). 

Then I had the epiphany for applying the classic Big Data framework to GDS. 
Most teams treat a golden dataset like a spreadsheet they “should probably have.”

In AI Agent world, a golden set is something else:
* It’s your definition of “good”.
* Your release gate.
* Your anti-regression insurance.

The “big data” thinking - the 6V's - is a solid approach to building high quality GDS. 

1) Volume — Do we have enough test cases?
2) Variety — Do we cover different kinds of reality?
3) Velocity — How fast does it go stale?
4) Veracity — Can we trust labels + scoring?
5) Value — Does the dataset protect what matters?
6) Variability — Do we capture ambiguity + distribution shift?

The MAS twist: Golden sets are two-layer
If you’re building multi-agent systems, the 6Vs naturally imply two eval views:
✅ System-level E2E golden set (user journey → final outcome)
✅ Per-agent golden sets (tight contracts → faster debugging + safer iteration)

One last point: you don’t always need massive GDS. Often, a small pilot (~20) tells you the score variance, and you can estimate an effective test size—sometimes ~100 is plenty, not 5,000.

I have covered this in detail in my newly published book "The Illustrated Guide to Multi-Agent Systems" - link in comments. 

#EvalOps #LLMEvaluation #AgenticAI #MultiAgentSystem

Yet another awesome session from SuperInterviews with Kasey Superinterviews and Jeremy Barton. 

Hear Deborah Liu say there is no such thing as true failure in interviews. Every interview is a prep to the next interview.

I am convinced that simulation based learning and exploration should be an integral part of AI agent building. 

I used to learn multi-agent systems like this:
📖 read an article, blog, paper, course material
🧠 seem to get it and nod my understanding
🧩 imagine how I can implement it into my product
🚀 vibe build a MAS prototype
😅 get surprised when I make it work for real 

The uncomfortable truth: I could design and vibe build MAS architecture… but I didn’t have full confidence in my system design until I could see the system behave under real conditions.

So I changed how I learn.
Now I learn MAS architecture like a pilot learns flight:
In a simulator.

Not “animated diagrams.”
A decision lab: inputs → choices → consequences → artifacts.
I pick a simple multi-agent use case (any domain works: support, finance, research, ops) and simulate the build approach and design decisions. 

This is what’s helping me become a Super IC AI PM — because the simulator makes the invisible… visible:

Tradeoffs stop being theoretical
Topologies aren’t “better/worse.” They’re trade spaces.
You can literally watch coordination tax show up, and see where the quality gains stop being worth the latency/spend.

Reliability becomes a systems skill
Not prompt luck.
I inject failures (noisy retrieval, flaky tools, partial context, conflicting agents) and learn what actually stabilizes systems: guardrails, budgets, circuit breakers, fallbacks, escalation paths.

Production thinking becomes muscle memory
EvalOps, observability, and safe rollout aren’t just checklists — they’re behaviors over time.
Shadow → Canary → Full becomes something you practice, not just plan.

My litmus test is simple:
If a simulation helps me make a better build and decision decision and teaches me why, it’s real learning.

Here are three examples of simulation based artifacts that helped in better MAS system design: 

1. The EvalOps & Observability Dashboard is an interactive simulation that implements the complete evaluation lifecycle for multi-agent AI systems. Manage golden test sets, monitoring real-time metrics (latency, accuracy, cost, safety) and debug with distributed traces. 
https://lnkd.in/gvEE67RY

2. The Failure Mode Diagnostic Lab is an interactive simulation that shows the four critical failure modes in multi-agent AI systems: Loops, Tool Thrashing, Cost Explosions, and Security Breaches. Instead of just reading about what goes wrong, you can watch failures cascade through real-time metrics and audit logs—then apply fixes and see them work.
https://lnkd.in/g2j6SbjZ

3. The Cost Explorer is an interactive simulation that makes multi-agent system costs tangible. It visualizes the five root causes of cost explosions—context snowballing, unlimited fan-out, missing caches, memory hoarding, and wasted verifications. 
https://lnkd.in/gBnxx7qt

Simulation Artifacts for Multi-Agent Systems Design

The Evolution of Evaluations

The evolution of evaluations is happening in plain sight:  we’re moving from LLM-as-a-Judge → Agent-as-a-Judge.

This recent paper - A Survey on Agent-as-a-Judge [https://lnkd.in/dh9ThVRR] is an awesome recent survey publication and captures the trend thinking quite well. 

Many people have commented and written about the paper albeit in a static commentary way but not delving into the nuances. With Google Dynamic views in Gemini, I have found a way to really understand an AI paper and understand the principles and working outlined in a paper. Instead of passive reading (or even “paper → code”), I’m building interactive simulation-based learning artifacts: https://lnkd.in/d42i3hgp

In this simulation based learning artifact, I simulate the judge’s workflow (plan → evidence → verify → deliberate → report), stress-test edge cases and failure modes, run “what-if” scenarios to expose hidden assumptions
and walk away with a mechanistic understanding of how this would behave in production. I have used a code generation use case to illustrate the difference between Agent-as-a-Judge and LLM-as-a-Judge. 

Please do check out Google Gemini Dynamic View artifact link above. 

For agentic evaluation specifically, simulation stimulated my thinking about: 
 What tools? What verification steps? What memory? What arbitration? What’s the stopping rule?

LLM-as-a-Judge was a huge leap: scalable, “near-human-ish” scoring, and a practical proxy for preference/reward signals. 

But it’s starting to hit a ceiling as evaluands become multi-step, tool-using, and domain-specialized. 

Where LLM-as-a-Judge breaks down
A single-pass judge tends to struggle with:
* Biases baked into the judge (e.g., rewarding verbosity, style, or its own output patterns). 
* “Looks correct” vs “is correct”: it often can’t verify claims against the real world, so you get hallucinated correctness. 
* Cognitive overload when the rubric has many dimensions → coarse scores that miss nuance. 
* Over-focus on final output, while failing to verify intermediate actions in complex workflows. 

What Agent-as-a-Judge changes
Agent-as-a-Judge turns evaluation into an active process, not a static opinion:
* Plans + decomposes evaluation into subtasks
* Uses tools to ground & verify (search, code execution, environment checks)
* Leverages multi-agent collaboration to reduce single-model blind spots
* Uses memory to preserve intermediate evidence and produce more traceable judgments 

The paper even frames a maturity curve:
Procedural → Reactive → Self-Evolving judge agents (from fixed workflows, to adaptive routing/tool use, to agents that can refine rubrics and learn over time).

If you’re building EvalOps for agentic systems, I think this shift matters:
the Judge is no longer a prompt — it’s an Agent system.

Like many others, I too am on the bandwagon of AI agent / AIoT agents. 
I have delved into the nitty gritty of ~10 multi-agent implementation - 2 in Healthcare, 1 in AI Infra and the rest across different use cases mostly for productivity and product management. 
One of the MAS is currently deployed and running in a Hospital system. 

From these experiences, I’ve learned the hard way that with multi-agent systems, “it works” in a demo doesn’t mean it will survive production.

The biggest gap isn’t tooling. It’s intuition—how agents behave over time when tools fail, handoffs break, loops run, costs spike, or security boundaries get fuzzy.

So I’m sharing a systems-thinking approach I’m excited about: Simulation-First Development. Before we prompt and build, we simulate. We practice the tradeoffs. We inject failures safely. We generate artifacts that become design records.

In the article below, I break this down and outline a 22-simulation roadmap that mirrors the full multi-agent lifecycle—from mental models all the way to EvalOps, scaling, and enterprise constraints.

I have documented the complete set of 22 simulation based learning and exploration artifacts that are enumerated in the roadmap described in the article. 

If you would like to take a look at these simulation artifacts, comment SIM below and I shall send you the link.

A Systems Thinking Approach to Vibe Engineering Multi-Agent Systems 85% of AI agent pilots never reach production. Not because the technology failed. 

Open article: Simulation-Based Multi-Agent Development by Harsha Srivatsa • 4 min read

taplio

The Battle

Jeff Frick

Harsha Srivatsa

Followers

Posts

Engagements

Likes

Performance ScoreJeff Frick's LinkedIn Performance

Performance ScoreHarsha Srivatsa's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Jeff Frick

Harsha Srivatsa

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Jeff Frick's LinkedIn Performance

Performance ScoreHarsha Srivatsa's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreJeff Frick's LinkedIn Performance