PandaProbe

Where does PandaProbe sit relative to LangSmith, Langfuse, and Helicone? They all claim "agent observability" but mean very different things underneath — some are basically prompt loggers, others actually trace tool-call DAGs. Curious which problem you decided was the real one.

Evaluation is the hardest part of this whole space and most platforms hand-wave it. The failure mode that actually bites in production isn't crashes or schema errors. It's slow drift in subjective quality (voice, classification accuracy, output style) that only shows up when a human reads 50 outputs in a row. How does PandaProbe handle that in practice? LLM-as-judge with custom rubrics, human-in-loop on a held-out set, embedding-distance from a golden corpus, or something else? And how do you stop eval cost from outpacing inference cost when you're re-judging every trace?

Great pain to tackle, Sina. Good luck.

Handling state and debugging for long-running autonomous agents is usually a nightmare, so having an open-source platform to standardize that workflow is huge. I can definitely see myself using PandaProbe to self-host my agent evaluation pipeline to keep sensitive client data entirely local. I am really curious to hear if you currently support custom tracing for raw API calls instead of just the standard frameworks.

Really nice work. The gap between "it ran" and "I understand what happened" is enormous for agents and nobody's solved it cleanly yet. Rooting for you!

Congrats on the launch and thanks for using mcp-use :)

Congrats on another great product going live! does it support MCP tool tracing natively or do you have to instrument those calls manually?

👋 Hey Product Hunt!I’m Sina, founder of PandaProbe.Building AI agents is getting easier, but understanding and trusting them in production is still hard.Once agents start calling LLMs, tools, APIs, MCPs, and sub-agents, logs aren’t enough anymore. You need to see what happened, why it failed, whether quality regressed, and how reliable the system is across full sessions.PandaProbe is my attempt to solve this: an open-source agent engineering platform for tracing, evaluation, monitoring, and debugging AI agent applications.The goal is simple: help developers move from “it works on my laptop” to “I understand production behavior, can measure quality, and continuously improve it.”What PandaProbe provides🔎 Trace — capture full agent executions as sessions, traces, and spans across LLMs, tools, agents, and custom logic.📊 Evaluate — score traces and sessions using mission-critical, agent-specific metrics.⏱️ Monitor — schedule recurring evaluations to automatically validate new traces and sessions in production.📈 Analytics — track performance, cost, latency, errors, and quality trends over time.🛠️ Open source + cloud — use the open-source core on GitHub or run PandaProbe in the cloud.Who it’s for🧑‍💻 AI engineers — debug agent behavior across LLMs, tools, and workflows.🏗️ Platform teams — monitor quality, regressions, and reliability in production.🔬 Builders experimenting with agents — understand failures and iterate faster.🚀 Startups — add observability and evaluation before things become unmanageable.reason about.Quick linksGitHub: https://github.com/chirpz-ai/pandaprobeDocs: https://docs.pandaprobe.comCloud: https://www.pandaprobe.com/I’ll be here all day answering questions and collecting feedback.If you’re building agents today, what’s the hardest part to debug or evaluate?Thanks for checking it out 🙏— Sina