SaaS Metrics
Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.Leaderboard right now: #1 - Jev 74.4
#2 - SemIf 73.1
#3 - djev 73.0
#4 - Winnow-12B Q8 71.2
#5 reflex 4B 70.3.
MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes:https://github.com/fstandhartinger/jevbenchTwo no-signup demos:https://who-is-right.app.mintapis.comhttps://is-it-ai-slop.app.mintapis.comLimitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.Wdyt?
Show HN: JevBench, a reproducible benchmark for typed decision models
No tagline provided.
View Origin LinkProduct Positioning & Context
Related Ecosystem & Alternatives
Discover adjacent products, open-source repositories, and developer tools sharing similar technical architecture.
Deep-Dive FAQs
What is JevBench, a reproducible benchmark for typed decision models?
JevBench, a reproducible benchmark for typed decision models is a digital product or tool described as: Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perfor...
Where did JevBench, a reproducible benchmark for typed decision models originate?
Data for JevBench, a reproducible benchmark for typed decision models was aggregated directly from the Hacker News community ecosystem, providing a stored public-discussion sample, not a representative customer survey.
When was JevBench, a reproducible benchmark for typed decision models publicly launched?
The initial public indexing or launch date for JevBench, a reproducible benchmark for typed decision models within our tracked developer communities was recorded on September 23, 2026.
How popular is JevBench, a reproducible benchmark for typed decision models?
JevBench, a reproducible benchmark for typed decision models has recorded public engagement of 154 traction score and facilitating 39 recorded discussions or engagements.
What are some commercial alternatives to JevBench, a reproducible benchmark for typed decision models?
Text similarity retrieved a product for further investigation: Fleeep. Functional substitutability has not been verified.
How does the creator describe JevBench, a reproducible benchmark for typed decision models?
The original author or development team describes the product as follows: "Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.Jev-class models return bounded ch..."
Community Voice & Feedback
Are there any "System One" models or jev like models that run on ollama?
Can you please add our model to benchmark?
https://gambler-relay-us-west1.leo-fish.ts.net/demo
https://gambler-relay-us-west1.leo-fish.ts.net/demo
How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?
It is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.
I was trying to figure out what exactly the tests here are. I guess I found some of the questions (here: https://github.com/fstandhartinger/jevbench/blob/main/datase...)e.g., "instructions": "Which intent does the user's message express?",
"labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
"state": "Play some Taylor Swift.",
"expected": "play_music"
"labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
"state": "Play some Taylor Swift.",
"expected": "play_music"
You can spot vibecoded websites by how they include the prompt or commit-style comments into the literal interface, instead of communicating it via visual context (or simply excluding it)> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
$40m in funding, 2 years in stealth.Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?Is it surprising that Jev consistently thinks it's Qwen?I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-ind... and the results do not seem to add up and also model sets are different... still needs time to mature likely
jev ceo on why he eschewed benchmarking: https://www.latent.space/i/216783460/privacy-benchmarking-an...
Discovery Source
Hacker News Aggregated via automated community intelligence tracking.
Tech Stack Dependencies
No direct open-source NPM package mentions detected in the product documentation.
Media Tractions & Mentions
No mainstream media stories specifically mentioning this product name have been intercepted yet.
Deep Research & Science
No direct peer-reviewed scientific literature matched with this product's architecture.