← Back to AI Insights
Gemini Executive Synthesis

The `harness2-judge`'s grade persistence logic during task re-runs and judge failures, specifically the `_merge` operation's handling of run IDs and rollouts.

Technical Positioning
Ensuring data integrity and accurate reporting within the `harness2` system, particularly concerning task re-runs and judge reliability. The system must reflect the current state of a task, not a stale one.
SaaS Insight & Market Implications
This issue exposes a critical data integrity flaw within `harness2`. The system's `_merge` logic incorrectly prioritizes stale graded data over new, potentially ungraded, rollout states when a re-judge operation fails. This leads to reports displaying scores for non-existent deliverables, undermining the reliability of performance metrics. For B2B SaaS, such data inconsistencies erode user trust and complicate debugging, particularly in automated evaluation pipelines where transient judge failures are inevitable. The core problem lies in the fixed run ID paradigm clashing with dynamic rollout updates, demanding a more robust state management and merge strategy to ensure accurate, real-time reporting.
Proprietary Technical Taxonomy
harness2-judge judged.json run ids rollout _merge partial task re-runs DemoAdapter judge CLI

Raw Developer Origin & Technical Request

Source Icon GitHub Issue Oct 18, 2026
Repo: google-research/harness2
harness2-judge keeps an old grade after a task is re-run if the re-judge fails

## Description

If a task is re-run and the next `harness2-judge` call fails for that task, `judged.json` keeps the grade of the rollout that was replaced. The report then shows a score for a deliverable that no longer exists.

## Problem

Run ids are fixed labels (`//`), so a re-run task writes its new rollout under the same run id. A partial task re-runs on resume, for example. `_merge` takes "same run id" to mean "same rollout". When the new judge call fails, it keeps the old graded row in place of the new ungraded one.

This repro uses `DemoAdapter`, a judge that grades the answer on disk, and the judge CLI's own collect / judge / merge / write steps. No network is used:

```
attempt 1: partial # sequential mode hit a transient 429
judge 1: demo-oc-agg1/ledger-1/agg1 1/1
attempt 2: ok # resume re-runs the task; the new rollout is wrong
on disk: demo-oc-agg1/ledger-1/agg1 -> 'Total: 105.25'
judge 2: demo-oc-agg1/ledger-1/agg1 1/1 graded=True # judge was down; old 1/1 kept
```

Repro script

```python
import json, tempfile
from pathlib import Path
from unittest.mock import patch

from harness2.cli import judge as JCLI
from harness2.core import judge as J, recursion, schedule
from harness2.core.proposer import Reply
from harness2.demo import DemoAdapter, DemoProposer

class Adapter(DemoAdapter):
judge_down = False

def score(self, run_...

Developer Debate & Comments

No active discussions extracted for this entry yet.

Adjacent Repository Pain Points

Other highly discussed features and pain points extracted from google-research/harness2.

Extracted Positioning
A `--max-usd` spend limit feature for `harness2` runs, designed to control monetary costs associated with model-backed experiments.
Enhancing cost governance and predictability for users running resource-intensive, model-backed experiments. This positions `harness2` as a more financially responsible and enterprise-ready platform by preventing unexpected expenditure.

Frequently Asked Questions

Market intelligence mapped to The `harness2-judge`'s grade persistence logic during task re-runs and judge failures, specifically the `_merge` operation's handling of run IDs and rollouts..

What problem does The `harness2-judge`'s grade persistence logic during task re-runs and judge failures, specifically the `_merge` operation's handling of run IDs and rollouts. solve?
Based on our AI analysis of the original developer request, its primary technical positioning is: Ensuring data integrity and accurate reporting within the `harness2` system, particularly concerning task re-runs and judge reliability. The system must reflect the current state of a task, not a stale one.
Which technical concepts are associated with The `harness2-judge`'s grade persistence logic during task re-runs and judge failures, specifically the `_merge` operation's handling of run IDs and rollouts.?
Our proprietary extraction maps The `harness2-judge`'s grade persistence logic during task re-runs and judge failures, specifically the `_merge` operation's handling of run IDs and rollouts. to adjacent architectural concepts including harness2-judge, judged.json, run ids, rollout.

Engagement Signals

0
Replies
open
Issue Status

Cross-Market Term Frequency

Quantifies the cross-market adoption of foundational terms like _merge and rollout by tracking occurrence frequency across active SaaS architectures and enterprise developer debates.