The `jevlike` project's evaluation methodology, specifically the `shuffle_context` control mechanism used to measure model performance lift in context-aware scenarios like Wikispeedia navigation.

`metrics(..., shuffle_context=True)` rolls contexts by one inside each batch (`model.py`, `context.roll(1, dims=0)`), so each menu is scored against the previous row's page. The Wikispeedia builder buckets rows by target article and writes them in path order, so neighbours share targets: on the test split, 39.4% of shuffled partners carry the same `Target article:` line, and 5.8% the same current page. A random permutation within the batch would leak 7.2%. Script and numbers: github.com/collapseindex/din... (`control_leak.py`, run against the JSONL your own `scripts/get_wikispeedia.sh` builds, at `94f5fd1`).

On a from-scratch checkpoint (tiny encoder, 3 epochs, seed 42, included in that folder) the control reads 14.9% top-1 on the full test split. The same checkpoint with the context replaced by an uninformative stub and the options kept reads 5.0% [0.04, 0.07] on 1,000 seeded items (uniform floor 3.6%); informed it reads 29.8% [0.270, 0.327], which matches the README's ~29%. So the control reports roughly half of the model's real lift over "no page". The model is doing the thing; the control is what's off, in the direction of understating you.

Two cheap fixes, and reporting both would say more than either: permute contexts across the whole evaluation set rather than within a batch, and add a blank-context arm. They answer different questions ("wrong page" vs "no page").

Everything above re-derives from the folder linked: pod spec, a ten...