Caliper, a local and lightweight harness for reliability testing of LLM skills, providing a pass@k score.

Skills for Claude Code and Codex are hard to test. What I mean by hard is that there's no standard way to do it. You evaluate the skill once on something, it looks like it works. You publish it. Then the new super model releases (GLM 5.2 anyone?), it will quietly break for some part, and you won't find out until your users complain.I also faced the same problem, so I tried to build something lightweight to stop doing that. Caliper.It's a local and lightweight harness that runs a skill k times in isolated environments and gives you a pass@k score (How much times it succeeded in these k times). As a non-deterministic technology, you can't just say "it worked once". You need to answer how much it passed in k times.You define success in a YAML spec. I picked YAML to keep a schema and make it still readable for a human. You either use a LLM judge, a Python assertion, or both:Here's an simple evaluation example with a JSON extraction, so you write this in a YAML file: tasks:
- name: Extracts action items as clean JSON
prompt: "Read /tmp/transcript.txt and write the
action items to /tmp/actions.json."
expect: "A valid JSON array where every item has
owner, task, due. No markdown fences."
assert: |
import json
items = json.load(open("/tmp/actions.json"))
assert isinstance(items, list)
assert all({"owner","task","due"}