Findings

Pre-registered evaluation results from the Role Drift Environment. Numbers trace to published artifacts — not headline claims.

V10 · June 2026

Train/eval disconnect

GRPO on composable drift detectors optimized training return to a peak group-mean of +3.22 (episode 98, V9 run). Held-out in-domain eval on 10 scenarios × 5 seeds falsified the deployment hypothesis: trained checkpoint mean return −1.58 [−1.84, −1.34] vs prompted baseline +0.19 [−0.19, 0.55]; n=50 each. Confidence intervals do not overlap.

Transfer eval on DearConnect scenarios (H2) also falsified: trained −0.82 vs baseline +0.48. Reward-hacking probes falsified: trained policy scored below all four trivial policies on eval (best trivial: −0.07).

What this means

Detector calibration is necessary for learnable training signal (uncalibrated detectors produced negative learning slope). It is not sufficient for held-out improvement under this recipe — rollout length mismatch, distribution shift, and reward gaming are plausible contributors, not yet isolated.

The environment and eval harness work as intended: they falsified a deployment win that training metrics alone would have suggested. That is the point of building measurement before scaling.

Artifacts