RLVR vs RLHF
01 · RLVR · Why

The Agent Core's job is a policy decision, not a sentence

At every step it picks one of four moves: call a tool, load a skill, ask the user, or return a final answer. Each of those has an outcome the runtime can check — so the training signal doesn't need a human in the loop.

Four structured moves, every step

  • tool_call — invoke one of the 19 tools
  • skill — load one of the 22 skills
  • AskUser — request clarification
  • final_answer — stop and reply

Why that's verifiable, not subjective

  • Each move produces a fact: a matched/mismatched type, a tool that ran or threw, a checklist that's satisfied or not.
  • The runtime and the recorded trace already contain the answer — grading is a lookup, not a taste call.

Where RLHF would struggle here

  • Needs a human (or a preference model trained on humans) to rank completions — slow, costly, and noisy at RL scale.
  • GRPO needs G≈8–16 scored completions per state, across thousands of states — a per-comparison human cost doesn't fit that volume.
  • Preference labels drift and disagree with each other; a schema check or an exit code doesn't.
02 · RLVR · Comparison

Same RL loop, a different place the reward comes from

Both feed a scalar reward into the same GRPO update. What changes is who — or what — produces that scalar.

RLVR — verifier in the loop

completion = policy.act(state)

reward = verify(completion, trace, env)
# reward ∈ {-1, 0, +1} per dimension,
# summed into one scalar

# deterministic: same completion always
# scores the same
# cost: one function call, ~free
# scales to G x N_states with no queue

verify() is code: schema checks, exit codes, trace lookups — no model call.

RLHF — preference model in the loop

A, B = policy.sample(state, k=2)

label = human.prefer(A, B)
# or: reward_model.score(A, B)
#     (itself trained on human labels)

# noisy: raters disagree with each
# other and with themselves
# cost: $ per label, or a reward
# model that can be gamed
# scaling to GRPO's G x N_states
# means either a lot of raters or
# a lot of drift

Standard for judging taste, tone, or writing quality — not what's being judged here.

03 · RLVR · Verification case 1

Correct action type

The coarsest check: did the model reach for the right kind of move at all, before anyone looks at which tool or what arguments.

Verify: action type match

# user: "What's 15% of 240?"
gold.action_type = "final_answer"   # no tool needed

pred.action_type = "tool_call"      # model reached for Bash

reward = +1 if pred.action_type == gold.action_type else -1
# -> -1: reached for a tool on a question
#    that needed no execution at all

gold.action_type is read straight off the recorded trace — a single string compare against one of four values.

04 · RLVR · Verification case 2

Correct tool or skill selection

Given the action type is a tool call, did it pick the right one out of 19 tools and 22 skills?

Verify: tool/skill match

# user: "Make me a logo for a coffee shop"
gold.tool = "ImageGenerator"
gold.equivalents = {"ImageGenerator"}   # no valid alt here

pred.tool = "VideoGenerator"

reward = +1 if pred.tool in gold.equivalents else -1
# -> -1: wrong generator family entirely

# elsewhere, equivalents can be >1 wide:
# gold.equivalents = {"Bash", "Glob"}   # "find" vs Glob, same result

A small acceptable-alternatives set per gold case keeps a stylistically different but equally valid pick from being punished.

05 · RLVR · Verification case 3

Valid tool arguments

The right tool with the wrong arguments still fails the task — so arguments get their own check, run before the tool ever executes.

Verify: argument schema + sandbox

# ShareFile requires {file_path, recipient}
pred.arguments = {
  "file_path": "reports/q3.pdf",
  "recipient": "finance@company.com",
}

ok = (
  json_schema.validate(pred.arguments, ShareFile.schema)
  and sandbox.path_exists(pred.arguments["file_path"])
)

reward = +1 if ok else -1
# -> catches missing fields, wrong types, and
#    paths that don't exist in this session's sandbox

The same schema and filesystem the tool itself needs to run — verification adds no new source of truth.

06 · RLVR · Verification case 4

Successful execution of the selected action

Right tool, valid arguments — did it actually work when the sandbox ran it? This check is independent of whether the earlier choices were right.

Verify: runtime outcome

result = sandbox.execute(pred.tool, pred.arguments)

reward = +1 if result.exit_code == 0 else -1

# Bash: "pip install pandas"    -> exit 0   -> +1
# Bash: "cat missing_file.txt"  -> exit 1   -> -1
# WebFetch: 200 response        -> success  -> +1
# WebFetch: connection timeout  -> error    -> -1

The runtime is its own verifier here — no gold label needed, just whatever the sandbox actually returns.

07 · RLVR · Verification case 5

Appropriate stopping behavior

Two failure modes in opposite directions: stopping before the task is actually done, and continuing to call tools once it already is. Both are trace bookkeeping, not judgment.

Verify: premature stop vs. unnecessary loop

# Task: Read file -> transform -> Write result
completed_steps = {"read"}            # model stopped here
required_steps  = {"read", "write"}

premature_stop = bool(required_steps - completed_steps)
reward = -1 if premature_stop else +1
# -> -1: gave a final answer before writing the output

# Loop check: same call signature repeated
history = [hash(c) for c in trace.tool_calls]
looping = history.count(history[-1]) > 2
reward -= 1 if looping else 0
# -> penalize re-issuing an identical call with
#    no new information since the last attempt

required_steps comes from the task's declared sub-goals; looping is detected from call-signature repetition in the trace.

08 · RLVR · Verification case 6

Overall task completion

The composite check: does the full trajectory actually finish what the user asked for — the metric that matters most, built from the same boolean facts as the other five.

Verify: end-to-end checklist

# user: "Email the quarterly report to finance@company.com"
checklist = [
  any(c.tool == "ShareFile"
      and c.arguments["recipient"] == "finance@company.com"
      for c in trace.tool_calls),
  trace.tool_calls[-1].result.exit_code == 0,
  "sent" in trace.final_answer.lower(),
]

reward = +1 if all(checklist) else -1
# -> one missing leg -- wrong recipient, a failed
#    send, or a final answer that doesn't reflect
#    it -- and the task did not complete

Every entry in the checklist is one of the earlier verifiable checks, composed against the specific task's goal state.

09 · RLVR · Composition

Six deterministic checks, one scalar reward

GRPO needs a single reward per completion. Each verifiable dimension above contributes its own term; nothing here waits on a human or a preference model.

What RLVR buys

  • Scalable — a function call per completion, not a rater queue; fits GRPO's G≈8–16 rollouts across thousands of states.
  • Consistent — the same completion always scores the same; nothing to disagree with itself over.
  • Cheap — reuses the schemas, sandbox, and trace the tools already need to run.

Where the honest limit is

  • These six checks cover routing, arguments, execution, stopping, and completion — not whether a *skill's* decomposition was sensible, or whether stopping here was the better call among several valid ones.
  • Those genuinely unverifiable dimensions still need a judge — see the Generalised Plan's Nemotron reward-judge slide, kept deliberately separate from what code already verifies.