Agentic RL 零基础教程

The default data is a tutorial-synthetic set and the repo says so as a release boundary: swap in real data and fix the evaluation protocol before publishing any business conclusion. It also reports a case where the keyword-rule baseline hit F1 1.0000, above both models, which the author reads as proof that no model was needed at that dataset difficulty.

Its position is that single-step, closed-set judgment can be externalized to a decision model or a rules engine, while multi-step trajectories, credit assignment and orchestration still need Agentic RL. It also argues an external judge cannot sit inside a training loop: reward functions are called millions of times on a millisecond budget, against 70-500 ms answers.

minimal-verl trains Qwen2.5-0.5B on Chinese sentiment three-class with TRL's GRPOTrainer and supports MPS; cd minimal-verl, uv sync, then uv run python scripts/00_check_env.py runs a five-step check. The decision bench is separate: make main takes about 7 minutes 40 seconds, make cv about 40 minutes.

Not from this bench alone. Sweeping 101 thresholds left a Pareto front with a single point, the encoder-only route, so the cascade bought nothing measurable. The author also shows the conclusion flips with dataset difficulty, which is the reason to treat it as one measured setup rather than a general result.

Three routes on one schema, one dataset and one metric set: a keyword rules_v0 baseline as the floor, an encoder-only Chinese MacBERT at 102M parameters, and Qwen3.5-0.8B with LoRA as the LLM control. The project ships 13 scripts, 33 reports, 32 tests and four figures, with make main running the whole chain in about 7 minutes 40 seconds.

Statistically indistinguishable on quality: five-fold cross-validation intent F1 was 0.9058 for the encoder against 0.8967 for the LLM, a gap of 0.33 standard deviations. Cost is where they split, at 20.67 ms versus 142.51 ms end-to-end P50 and 390.6 MB versus 1728.1 MB deployed.

Both, under one MIT license. Thirty-three Chinese Markdown chapters run from RL basics through PPO, DPO and GRPO to a chapter on when judgment belongs in a decision model rather than in training, and minimal-verl trains Qwen2.5-0.5B on Chinese three-class sentiment with TRL's GRPOTrainer over an MPS-capable path.