Prefactor

Using this currently to secure our agents, absolutely shocking to me that people give the keys to the kingdom to their AI without considering how this all plays out. Love this product.

The evals pass then production drifts problem has a close cousin in ordinary end to end testing, where a suite goes green on a dev laptop and red on a slower CI box. Same root cause both times: the check was written against conditions that only held in one environment.What I would want to know here is how you handle the scoring being non deterministic itself. If the judge is a model, its verdict drifts too, so a regression alert could mean the agent changed or the scorer changed. Do you pin the scorer version and replay old runs through new scorers to separate those two?Congrats on the launch.

This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.

Congrats on the launch! How do you evaluate the AI Agents is it deterministic or using LLMs by providing the truth table?

Congrats on launching! The "did it actually do it, or just say it did" framing will ring true for anyone who has put an agent in front of real users. The 25k spans a month free tier makes it easy for a small team to give this a proper try. How do the custom spans that pull context from Linear or GitHub work in practice, do you write those checks yourself or pick from templates?

Matt, following on from the forum thread where I said the zero-token layer does most of the work: the thing I keep bumping into is that every deterministic check I own exists because something already went wrong once. Banned phrases, schema conformance, the contradicts-its-own-inputs assertion, all of it got written the morning after a bad output shipped. So my free layer's coverage is really a map of my incident history, and the failure it structurally cannot see is the one I haven't met yet.That interacts awkwardly with scoring 100% of traffic. Full coverage on rules-someone-wrote reads as all-green on the dashboard, and all-green on 100% is a much stronger-sounding claim than all-green on a 30% sample, which is the version everyone already knows to distrust.You told Yolanda the pattern layer asks how the agent's behaviour differs "from what you know to be the case", and that phrase is where my question lands, because that's still a declared baseline. Is there anything that flags a run as unusual without someone having said what usual is, off span shape, tool sequence, turn count? That's the piece that catches the first instance rather than the second, and it's the only part of this I've never managed to build for myself.

Congrats Matt, Simon and team, this thread is a masterclass! One angle nobody’s asked: your pitch says built for engineering teams, but at our company the agents are built and owned by ops and marketing people like me, not engineers. Once the SDK is wired up, can the business owner of an agent actually live in Prefactor day to day, reading quality scores and approving held runs without understanding what a span is? The 40-agents-no-honest-answer problem exists at non-engineering companies too, we just have nobody to hand the dashboard to.

congrats on the launch! curious as to what metrics they evalaute the other AI agents on

Real-time eval is the piece most agent stacks skip until something breaks in prod. Curious how you handle scoring when there's no ground-truth label — is it rubric-based, or are you comparing against a reference trace? Congrats on the launch.

The "tokenless until you actually need an LLM" design is the part that stands out, most eval tools default to LLM-as-judge for everything and eat the cost whether it's warranted or not. Given you're scoring 100% of production traffic, curious how Prefactor handles agents that call other agents (sub-agent chains). Does a breach in a nested sub-agent bubble up and pause the parent run too, or does each agent in the chain get evaluated as its own isolated instance?

That's exactly the gap that matters. An agent can produce a perfect transcript and still fail the task in the real world. We've seen jobs report "success" while nothing actually changed. That's why the real evaluation isn't what the agent said—it's what actually happened. Can Prefactor validate external outcomes, like confirming a page was updated, a record was written, or an API state changed, instead of only grading the execution log?

I really like the idea of tracking how the agent is working, as it is performing actions. Being more proactive is better than finding out something went wrong several days ago. It's not to be overlooked that the interruption isn't just killing the agent run, but handing it to a human too. I am going to have to play around with this and my own agents.

scoring the run live instead of charting it three days later is the part that actually matters. catching a bad agent after it already shipped the damage is just reporting.

One thing I'd love to see: does the scoring account for cost and latency alongside quality or is this purely about correctness? Teams I've talked to care about all three together, not in isolation.

The runtime controls are the strongest part here. Being able to hold or block an agent when an eval fails turns evaluation into an actual safety layer, not another dashboard.How do teams prevent a noisy or misconfigured eval from repeatedly stopping healthy production runs?