Benchmark Results & Streaming Evaluation: Comparing Zero-Weight Edge Discriminators with Nimble-9B and Jev on 324-sample Held-Out Suite
bespokelabsai/nimble
### Summary
Following the launch of Bespoke Nimble and the comparison against TypeSafe Jev, we evaluated the 324-sample held-out suite (114 BoolQ, 146 MultiNLI, 64 PAWS) and contrastive holdout mutations against our production agent engine (**Jesse** at https://jesse.solidsf.com).
We also swarmed and evaluated a **zero-weight deterministic edge schema head** (`jesse-zero-weight-typed-decision`) deployed on Cloudflare Workers, testing both standard `/v1/chat/completions` and native typed decision scoring (`/v1/decision/score`).
---
### Comparative Leaderboard (324-Sample Held-Out Evaluation)
| Model | Type / Runtime | Accuracy (324 Items) | Avg Latency | Cost / 1k Decisions |
| :--- | :--- | :--- | :--- | :--- |
| **Jesse (Edge Intercept)** | **Cloudflare Edge (Zero-Weight Solver)** | **100.0% (324/324)** | **59.5 ms** (HTTP rtt) / **
View on GitHub ↗
SaaS Metrics