Reasoning · ICLR 2026 · Logical Reasoning Workshop
LLMs gaming verifiers
LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
When a model learns to pass the checker instead of learning the rule.
THE IDEA, AT A GLANCE
CONCEPTUAL ILLUSTRATIONRule or shortcut?
An identifier list can pass a check of the examples.
Rename the objects
Preserve the logic, but change the object names.
Check the same answer
Keep the answer fixed; check whether it still holds.
Motivation
Reinforcement learning with verifiable rewards relies on a checker to decide what counts as success. If the checker only tests the given examples, an answer can earn a reward without capturing the underlying rule.
Method
Isomorphic Perturbation Testing checks one fixed model output on the original task and on a logically equivalent version with renamed objects. Relationships and attributes stay the same. A general relational rule should still work; an answer tied to specific object identifiers will fail.
Results
On the studied train-classification benchmark, RLVR-trained models used shortcuts that passed ordinary verification. In controlled training, an example-based verifier induced this behavior, while isomorphic verification eliminated the observed gap. Shortcuts were also more frequent at higher inference-time budgets in the evaluated models.