Skip to main content

BenchFlow — Architecture

The whole architecture, as one coherent picture — every concept we need, no build-order tiering. Release scoping and milestones are tracked separately (Linear). This document is derived from the sources that count — Han Lee’s writing and conversation, our project notes, and the agentic-RL literature — not from the current doc or the current code, both of which are snapshots that follow this, not the other way round.

What BenchFlow is

BenchFlow is the environment-and-rollout engine for agentic RL — it turns a stateful environment into evaluated, training-ready trajectory data, for any model and any trainer. It stops where the gradient starts. One engine, three modes. There is one thing — a scored rollout. Eval = score it and stop. Train = score it and hand the trajectory to a trainer. Monitor = score it in production. (Han Lee: “evaluation, reward and monitoring … it’s really all the same thing under different circumstances.”) The bet. A complete RL environment is E = {T, H, V, S, C} — Tasks, Harness, Verifier, State, Config (Han Lee, RL Environments for LLM Agents). BenchFlow targets the complete E. State management — stateful, multi-service environments that can roll out, roll back, and branch — is the frontier of agentic RL and the surface BenchFlow is built around. The boundary. BenchFlow owns environment + rollout + reward. Trainers own weights + gradients + optimizer. The trajectory is the seam — every RL trainer can consume BenchFlow output without coupling.

Grounding

This architecture rests on three sources, kept honest against each other. Han’s {T,H,V,S,C} (blog, RL Environments for LLM Agents) — the environment decomposition. Verbatim: T = “problems the agent tries to solve”; H = the agent harness, “scaffolding that … controls how the model interacts, but does not improve what it knows”; V = the verifier, “V: (task prompt, completion, info) → [0,1]”; S = state, “stateless (fresh starts) … or stateful (persistent across actions/episodes)”; C = configuration, “turn limits, context budgets, sampling temperature, curriculum scheduling.” Han’s conversation (advisory call) — the dynamics the blog’s static set does not capture:
  • “Environment 总是要 roll out, roll back”roll-out and roll-back are definitional for an environment. Roll-back = “snapshot environment and go back to its different stage.”
  • Branching: an ask_user-type interaction “literally is a checkpoint … to different type of rollout”, and from it — “From reward function to a value function … of the current state.” Branching is “very important for large horizon tasks.”
  • “eval = monitoring = reward” — one activity, observed across five spaces (output, action, reasoning, memory, latent).
  • “The harness is not meant to be intelligent” — self-improvement targets the model and skills, never the harness; “skill 是属于 memory” (skills are memory).
  • ACP is the mechanism for modelling human interaction inside a rollout.
The agentic-RL literature — agentic RL is a “temporally extended, partially observable MDP” (The Landscape of Agentic RL, 2509.02547) — definitionally a branching structure. Tree-structured rollouts give “step-wise process supervised signals even using only the outcome reward” and “more rollouts within a fixed budget of tokens or tool calls” (Tree Search for LLM Agent RL / Tree-GRPO, ICLR 2026). A scan of 13 RL libraries (verifiers, prime-rl, SkyRL, verl, NeMo-RL, Tinker, OpenEnv, Harbor, Terminal-Bench, Inspect, ORS, Gymnasium, agent-lightning) found all model rollouts linearly — so a tree-native rollout with environment snapshot/restore is genuine, defensible novelty, and the load-bearing hard part is snapshot/restore of heavy environment state (see “The hard part”).

Design principles

  1. The kernel depends only on contracts. The call graph is the source of truth; anything exported with no live caller is wired in or deleted.
  2. Four planes, each swappable, each managed + BYO — Sandbox, Agent, Environment, Reward.
  3. The Rollout is a tree. An RL episode is a tree of states; a linear rollout is the degenerate degree-1 case. Branch, snapshot, and restore are first-class — they are how a reward function becomes a value function.
  4. Roll-back is definitional. An environment that cannot snapshot and restore its state is incomplete (Han). snapshot/restore are real methods, not stubs.
  5. Zero-modification adoption. A benchmark brings a self-describing package + a manifest; it never subclasses BenchFlow or touches private APIs.
  6. Eval = monitoring = reward — one activity, scored on the same trajectory, across five spaces.
  7. The environment is a stateful state machine the framework provisions, snapshots, restores, and tears down.
  8. BenchFlow is the ACP Client — the “user” is a pluggable policy, not a special actor.
  9. The harness is not intelligent — its only job is to extract the most from the model; self-improvement targets the model and skills.
  10. Readiness and teardown are framework guarantees — never the benchmark’s burden.
  11. Ship beats design — a better design that doesn’t run loses to an adequate one that does.

The conceptual model — the planes

The kernel is three subsystems — Rollout lifecycle, reward, trajectory — importing only contracts/ (four Protocols). Concrete providers (Docker, ACP, ManifestEnvironment, RewardFuncs) join via a registry. The four planes map onto Han’s E: T and C are kernel concepts (the inputs); H, V, S are the planes that do the work; Sandbox is the substrate all three run on. Four planes, one kernel, two kernel-level inputs — that is the whole conceptual surface.

The execution model — tree-native

A Rollout is one RL episode, and it is a tree. Han’s trajectory is a chain of state → action → next-state; a Branch makes that chain a tree; classical RL is defined over exactly this tree (a POMDP), and the value function V(s) is defined as the expected return over the continuations from a state. Modelling the Rollout as a tree is therefore the RL-native choice. The execution model has three primitives, one derived view, and one authoring form — all defined on the one tree, not a Russian-doll hierarchy:
  • The primitives are irreducible. The tree (Rollout), its edges (Step — Han’s atomic unit; one Step is one “turn”), and the Branch operation (snapshot + fork). Branch is the credit-assignment engine: it evaluates one state across N continuations — averaging the children’s returns estimates V(s). That is Han’s “from reward function to a value function of the current state” and Tree-GRPO’s peer-reviewed result that a tree yields process supervision from a single outcome reward. A GRPO group run as a shared-prefix tree beats N independent rollouts (more rollouts per token/tool budget). Branches occur at ask_user-style interaction checkpoints (one child per option), at GRPO group points, and at value-estimation points.
  • Trajectory is a derived view — a pure function of the tree (a path), never declared. It is what serialises out: a linear prompt / completion / reward / metrics / info record, the Verifiers/ORS training unit.
  • Scene is authoring sugar — the declaration form for multi-phase / multi-agent rollouts (RolloutConfig.scenes). It desugars completely to per-Step role/skill attribution plus config that changes along the tree, and adds no expressive power. It has no runtime object and no lifecycle of its own — RolloutConfig.scenes is a desugaring pass that lowers to per-Step config. Kept only as a convenient authoring affordance. (The original RFC’s instinct — “a phase is just state” — was correct.)
Tree-native is free for the mental model, not for the engine. A linear rollout genuinely is a degree-1 tree, so the data model costs nothing extra for the common case. But the engine — checkpoint/fork, three-layer snapshot composition, node-addressed scoring, child scheduling — is paid for on day one even by users who only run linear rollouts. That is an accepted cost, not a hidden one: the tree is the correct foundation, and the linear path inherits its machinery.

Lifecycles

Every lifecycle the framework owns, as ordered phases. Job lifecycle. plan (resolve tasks × agents × repeats) → schedule (parallel-independent, or sequential-shared for continual learning) → run Rolloutsaggregatereport. Rollout lifecycle. setup (resolve config, build the environment object) → start (sandbox up) → provision environment (Environment plane starts services) → readiness gate (framework-guaranteed; the agent never runs before the world is healthy) → connect agent (ACP) → execute (the tree grows: Steps and Branches) → verify (Reward plane scores) → teardown. Branch lifecycle. quiesce (pause the agent at a stable point) → checkpoint (snapshot environment state, then container, then agent-session state — in that order, see “The hard part”) → fork (N children) → run childrenscore / aggregate (per-child return → V(parent)) → optionally restore the winning child’s state to continue. Environment lifecycle (Han’s roll-out / roll-back). provisionreadinessquery (expose state to the verifier) → snapshotrestoreresetteardown. snapshot/restore are definitional — the substrate every Branch runs on. Sandbox lifecycle. startexec / upload / download / expose_portsnapshot / restore (container-level, coarser than environment-state) → stop. A Rollout is checkpointable because three snapshot layers compose — container (Sandbox) ⊃ environment-state (Environment) ⊃ agent-session — but composing them correctly is a real consistency problem (see “The hard part”). The one store that deliberately does not roll back with a Branch is the continual-learning learner store (capability 5).

The four planes

Sandbox — where it runs. Compute substrate. Built-in: Local (raw Linux) + Docker. Optional: Daytona, Modal, Firecracker, K8s. BYO via the Sandbox protocol. Hardening (lockdown) is a capability flag. Framework-guaranteed readiness gate + teardown. An environment is declared once and runs on any provider. Agent — who acts. The agent under test (eval) or the policy under training — Han’s harness, “not intelligent.” Protocol: ACP (the official agent-client-protocol). BYO via --agent-import-path. The registry stores agent declarations as data, not install code in the kernel. A trainer-served policy endpoint (OpenAI-compatible, hot-swappable) is one agent provider type. The plane’s real surface is the Session (below) — not just connect.

Skill loading

BenchFlow treats mounted skills as agent-native memory, not prompt text. Skills are controlled by one run mode: no-skill, with-skill, or self-gen. no-skill hides any task-local environment/skills from the agent and strips that directory from copied build contexts. with-skill mounts the task’s environment/skills directory through the selected agent’s native skill paths. self-gen gives the creator scene only skill-creator; the solver scene sees only the generated skills root, never task-bundled skills. Advanced callers can provide a custom --skills-dir only in with-skill mode. Claude Code reads each skill’s frontmatter name and description for native discovery. The full SKILL.md body and bundled resources are loaded by the agent when it invokes or reads the skill; BenchFlow does not inline every mounted skill body into the task prompt by default. BENCHFLOW_SKILL_NUDGE is an optional prompt nudge layered on top of native discovery. Use name to tell the agent which skills are mounted, description to include each mounted skill’s description, or full to include the full SKILL.md body. Omit the variable to keep BenchFlow’s runtime default off. Environment — the world (Han’s S). The stateful world the agent acts in. Owns the world’s lifecycle: provision / readiness / query / snapshot / restore / reset / teardown. See “The Environment plane & the manifest.” Reward — how it’s scored (Han’s V). RewardFunc / Rubric / verifier. V: (task, completion, info) → [0,1], generalised to a graded, multi-space, multi-granularity signal over the trajectory tree. See “Evaluation.”

The four contracts

The kernel imports only these.
Session is part of the contract, not an untyped return — the entire ACP interaction (prompt, nudge, the ask_user branch hook) is the Agent plane’s seam, so it must be specified to BYO an agent. Reward.score takes a RolloutNode, and a node carries its tree context: node.path (root → node), node.subtree, node.state. One score method therefore expresses both outcome reward (read the leaf) and process reward (walk node.path across the Action and Reasoning spaces) — there is no per-step-in-isolation scoring. VerifyResult = {reward: float, items: dict[str, float], events: list[RewardEvent], space, granularity}.

The Environment plane & the manifest

What a benchmark author writes is the manifest — the entire integration surface. Write a manifest; your stateful environment runs anywhere and trains anything, with zero framework modification. The default adapter ManifestEnvironment reads it.
State is a real database; tools are read-write ops over the schema — which is what makes state snapshot-able, diffable, and verifiable. Two topologies behind one contract: in-sandbox (the environment runs in the rollout’s own sandbox — the default) and shared-fleet / sidecar (a long-lived service fleet + a TaskDatabase + AccountBroker for multi-tenant per-task accounts — the scale path). The Stateful Multi-Service Benchmark (SMSB). ClawsBench and chi-bench are structurally the same machine; the plane hosts both. ClawsBench is the internal dogfood (the manifest’s design partner); chi-bench is the external proof — a ~25k-LOC heavy simulator with a thin MCP transport, onboarded via a ~25-line manifest with its environment untouched, its ~920 LOC of Harbor coupling collapsing into the manifest.

Evaluation — the five spaces

eval = monitoring = reward. The same scoring runs at train time, at eval time, and in production — only the context differs. A reward signal is read from the trajectory across five spaces (Han): Every reward record is tagged (space, granularity, value). Granularity is terminal (the whole trajectory) or step (one edge) — an episode-level scalar alone is inadequate beyond ~50 steps; the tree’s structure supplies finer credit. Process reward is read by walking a node’s path across the Action and Reasoning spaces — not by scoring each step in isolation (process supervision “hard to judge” per-step — Han). The wire formats reward.txt / reward.json cross the sandbox boundary; the in-kernel model is VerifyResult + RewardEvent.

The interaction model — ACP

Human interaction is modelled through ACP’s role split: BenchFlow is the ACP Client; the “user” is a pluggable User Model inside the Client role. Two channels carry everything:
  • session/prompt (Client → Agent) — the task instruction and every nudge (user-initiated follow-up).
  • request_permission / ask_user (Agent → Client, with enumerated options) — agent-initiated, surfaced through Session.on_ask_user.
ask_user with enumerated options is the branchable interaction primitive — finite options ⇒ a finite, scoreable interaction tree (each option is one Branch child). The interaction tool is never hard-coded as “step one”; the agent chooses to use it, and the Action space scores whether it asked — an under-specified task makes “ask the user” the correct behaviour, and failing to ask is a negative reward (Han). User Model modes: scripted / simulated (LLM persona) / real-human / auto. (Branching is not a User Model mode — it is a property of the Rollout tree.)

The edges — adapters & trainers

The manifest is BenchFlow’s native format; adapters translate every other format to it. Inbound env adapters — Harbor, Inspect, ORS, PrimeIntellect/Verifiers environments → run foreign benchmarks natively. Terminal-Bench tasks run through the Harbor adapter (Harbor is itself terminal-bench-derived). Outbound — the trainer seam. A scored trajectory exports as a Verifiers / ORS JSONL record (prompt / completion / reward / metrics / info). Being a Verifiers/ORS-compatible producer yields a trainer — prime-rl — with zero trainer code. BenchFlow is a rollout service; trainers (Tinker, verl, NeMo-RL) stay external. The trajectory is the seam.

How a Task flows through the architecture

A Task (Han’s T) is the problem spec — one task.md plus the environment package and verifier. It is a kernel concept, and it is what wires the planes together for one run:
One Task → one Rollout tree → one or more Trajectories. A Job is many Tasks (or one Task × many repeats). {T,H,V,S,C} is not an abstraction layered on top — it is the wiring diagram of a single run.

The eight capabilities — how each fits

The architecture is one shape; these are the eight things it must carry. Capabilities 1–6 and 8 are benchmark-forced — “done” = that benchmark runs clean. Capability 7 is the substrate the others ride on, not a benchmark. All eight land on one architecture — four planes, a tree-native Rollout, an adapter edge. None requires a new top-level concept.

The hard part — honest risk

The library scan is unambiguous: no agentic-RL library ships environment snapshot/restore. Tree-GRPO branches a token prefix; Inspect’s fork() deep-copies conversation state, not the sandbox; its checkpoint system is resume-only — its design note says “reality doesn’t have a fork command.” BenchFlow’s bet is to branch a heavy stateful environment — a mock-Gmail SQLite database, a healthcare simulator, eventually a K8s cluster. The Branch checkpoint is genuinely three unsolved problems, not one:
  1. Environment-state snapshot — DB dump/restore, copy-on-write volumes, fork-able service processes. Designed deliberately, environment-class by environment-class — not one generic call.
  2. Agent-session snapshot — freezing and restoring a live ACP session (and the running agent process and its context behind it) is the same class of hard problem. Inspect can only deep-copy conversation state precisely because it cannot snapshot the process. The architecture does not get this for free.
  3. Cross-layer consistency — the three snapshot layers (container / environment-state / agent-session) have different consistency models; a naïve capture can produce a container snapshot and a DB snapshot that disagree about a write in flight. The Branch lifecycle therefore quiesces the agent first, then snapshots environment → container → session in order.
Tree-native rollout structure is proven and safe to commit to. The Branch checkpoint — all three layers — is the frontier: it is where the engineering risk concentrates and where the moat is, and it must be designed deliberately, not hand-waved as one call.

Adaptations — the decision log

So decisions are not re-litigated.

Appendix — research validation

Checked against the recent agentic-RL literature; the field’s shape matches.
  • Agentic RL = a temporally-extended POMDP — definitionally a branching structure. (The Landscape of Agentic RL, 2509.02547)
  • Tree-structured rollouts yield process supervision from a single outcome reward and more rollouts per token/tool budget. (Tree Search for LLM Agent RL / Tree-GRPO, ICLR 2026, 2509.21240)
  • All 13 surveyed RL libraries model rollouts linearly with no environment snapshot/restore — tree-native + heavy-environment snapshot is real novelty, not a reinvention.
  • Rollout-as-a-service decoupled from training; the trajectory is the seam. (ProRL Agent; PrimeIntellect Environments Hub)
  • Continual learning = a base policy + a persistent, evolving skill library; version and roll back the store. (MetaClaw, MemSkill, SkillLearnBench)
Verdict: the architecture is consensus-correct on shape and deliberately ahead of the field on one primitive — the Branch checkpoint (environment + agent-session + container snapshot) for stateful branching. The risk is execution of that primitive, not the design.