← Back to AI Insights
Gemini Executive Synthesis

JevBench

Technical Positioning
A reproducible benchmark for 'Jev-class' typed decision models, which are presented as disruptively faster and cheaper alternatives to LLMs for bounded choices and probabilities.
SaaS Insight & Market Implications
JevBench introduces a critical benchmarking tool for a new category of 'Jev-class' decision models, positioning them as superior to LLMs for specific, auditable decision-making tasks. The focus on accuracy, latency, and cost provides a comprehensive evaluation framework, directly addressing enterprise needs for performance and efficiency in automated decision systems (e.g., lending, fraud). This tool is vital for B2B SaaS providers developing or integrating such models, enabling transparent comparison and validation. The emphasis on reproducible benchmarks and open-source components fosters trust and accelerates adoption in regulated industries where model explainability and reliability are paramount. This directly challenges the LLM dominance in certain application domains.
Proprietary Technical Taxonomy
Jev-class models typed decision models LLMs bounded choices probabilities accuracy latency price

Raw Developer Origin & Technical Request

Source Icon Hacker News Sep 23, 2026
Show HN: JevBench, a reproducible benchmark for typed decision models

Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.Leaderboard right now: #1 - Jev 74.4
#2 - SemIf 73.1
#3 - djev 73.0
#4 - Winnow-12B Q8 71.2
#5 reflex 4B 70.3.

MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes:github.com/fstandhartinger/j... no-signup demos:who-is-right.app.mintapis.comhttps//is-it-ai-slop.ap... English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.Wdyt?

Developer Debate & Comments

joknoll • Sep 25, 2026
Are there any "System One" models or jev like models that run on ollama?
faangguyindia • Sep 23, 2026
Can you please add our model to benchmark? https://gambler-relay-us-west1.leo-fish.ts.net/demo
adityamishra241 • Sep 23, 2026
How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?
janalsncm • Sep 23, 2026
It is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.
ks2048 • Sep 23, 2026
I was trying to figure out what exactly the tests here are. I guess I found some of the questions (here: https://github.com/fstandhartinger/jevbench/blob/main/datase...)e.g., "instructions": "Which intent does the user's message express?", "labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "state": "Play some Taylor Swift.", "expected": "play_music"
dmix • Sep 23, 2026
You can spot vibecoded websites by how they include the prompt or commit-style comments into the literal interface, instead of communicating it via visual context (or simply excluding it)> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
hbrn • Sep 22, 2026
$40m in funding, 2 years in stealth.Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?Is it surprising that Jev consistently thinks it's Qwen?I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
sean_pedersen • Sep 22, 2026
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-ind... and the results do not seem to add up and also model sets are different... still needs time to mature likely
swyx • Sep 22, 2026
jev ceo on why he eschewed benchmarking: https://www.latent.space/i/216783460/privacy-benchmarking-an...

Frequently Asked Questions

Market intelligence mapped to JevBench.

How is JevBench positioned in the market?
Based on our AI analysis of the original developer request, its primary technical positioning is: A reproducible benchmark for 'Jev-class' typed decision models, which are presented as disruptively faster and cheaper alternatives to LLMs for bounded choices and probabilities.
Are engineers actively discussing JevBench?
Yes, we have tracked 39 direct responses and active debates regarding this specific topic originating from Hacker News.
What architecture is tied to JevBench?
Our proprietary extraction maps JevBench to adjacent architectural concepts including Jev-class models, typed decision models, LLMs, bounded choices.

Engagement Signals

154
Upvotes
39
Comments

Cross-Market Term Frequency

Counts stored text matches for foundational terms like LLMs and latency in analysed public discussions. These counts do not measure adoption.

Stored text matches, not adoption or market-share measurements.

  • LLMs: 61 records
  • latency: 29 records
  • price: 17 records
  • accuracy: 13 records
  • Cost: 5 records
  • Speed: 4 records
  • probabilities: 2 records
  • Jev-class models: 1 records
  • typed decision models: 1 records
  • bounded choices: 1 records
  • chance-corrected Intelligence: 1 records
  • Calibration: 1 records
  • MIT harness: 1 records
  • public items: 1 records
  • frozen artifacts: 1 records
  • scoring code: 1 records
  • per-task outcomes: 1 records