Project report · Bachelor's thesis in Computer Science
Preference-Guided Curriculum Discovery for Robust Reinforcement Learning
I trained a network to order game levels by difficulty from an LLM's pairwise judgments and used it to guide the training curriculum of a reinforcement learning agent. The learned ordering is informative. Whether it helps the curriculum depends on how the training loop uses it.
- Author
- Malik Durmus
- Institution
- LMU Munich, Chair of Mobile and Distributed Systems, 2026
- Period
- March to September 2026
- Stack
- Python · JAX · Flax · Optax · Hugging Face datasets · SQLite · FastAPI · Gemini on Vertex AI · Weights & Biases · Slurm · RunPod · LaTeX
The problem
Unsupervised Environment Design trains reinforcement learning agents on automatically chosen levels. The method I built on, Sampling for Learnability (SFL), scores a candidate level by p(1 − p), where p is the agent's success rate on it, and trains on the levels the agent solves about half of the time. At the start of training the agent fails almost everywhere, the score is zero for most levels, and the curriculum falls back to random sampling. I call this phase the cold start.
The thesis asks whether a difficulty ordering of the levels, learned before training from a large language model's comparisons of rendered levels, can supply the missing selection signal, and how it compares with making the empirical score itself denser.
How it works
What I built
Preference-labeling pipeline
Batch annotation of rendered level pairs with Gemini 3.5 Flash on Vertex AI: four datasets of about 22,000 pairwise difficulty judgments each, roughly 88,000 in total, with SQLite bookkeeping and compilation to Hugging Face datasets. A FastAPI tool for the human validation set of 690 pairs.
LLM agrees with the human annotator on 86% of decisive pairs
Difficulty model
A residual CNN in JAX/Flax trained with the Bradley–Terry pairwise loss, symmetry augmentation, five-member ensembles, and a rank-based normalization against a frozen reference sample, which makes scores from a pairwise-trained model usable as a target.
held-out pairwise accuracy 0.90 (JaxNav), about 0.80 (XLand-MiniGrid)
Curriculum integration
Extended the SFL codebase with a difficulty target that moves with the agent, four rules that combine the learned score with the empirical one, three controllers for the target, three dense learnability scores, a shortest-path heuristic, and a generator restricted to solvable levels. Ported the loop to XLand-MiniGrid.
about 200 training runs of 3·10⁸ environment steps
Evaluation and reproducibility
Ten paired seeds per method with exact Wilcoxon tests and Holm correction, a worst-case evaluation over 10,000 levels per policy, a measurement study of the cold start itself, and a scripted pipeline that regenerates every figure and result table of the thesis from the logged runs.
one command rebuilds all figures and tables
What the thesis shows
The network is informative. It reproduces the annotator's ordering on unseen pairs with up to 90% accuracy, the annotator agrees with a human on 86% of decisive pairs, and a curriculum guided by the learned ordering alone ends above one guided by a shortest-path heuristic.
Its value depends on how the loop uses it. The ordering places hard and unsolvable levels at the same end. With the original controller the difficulty target rises to its maximum, and the prior then pulls unsolvable levels into training. When the generator produces only solvable levels, the additive combination of the two scores is significantly better than SFL.
| Generator | Standard SFL | Linear Addition | Paired result |
|---|---|---|---|
| All levels | 0.980 | 0.977 | not significant (3 of 10 seeds ahead) |
| Solvable levels only | 0.986 | 0.991 | significantly better (9 of 10 seeds, p = 0.027) |
Findings in detail
- The cold start was not shortened. On JaxNav the prior gives a small early lead that disappears one cycle later and is not significant at ten seeds. The phase is short in training cycles but costs about 75,000 evaluation episodes per buffer refresh.
- The prior acts later in training. Few levels carry a non-zero learnability score, so SFL fills much of its buffer in random order. The additive rule keeps every level with a high learnability score and fills the rest with levels near the difficulty target.
- The target controller matters for what the agent trains on. The original controller keeps raising the target after the selected levels have become too hard. A controller that follows the agent keeps more solvable levels in training, but with four seeds it did not change the final win rate. Removing unsolvable levels was the stronger lever.
- The model alone is not enough. Selecting only by distance to the difficulty target stays below SFL with both generators, because a level near the target difficulty is not necessarily one the agent learns from.
- Denser empirical scores beat uniform sampling and two of the three finish above the difficulty model used alone, but none matches SFL's binary score.
- SFL's own score is noisy. In a second rollout of the same policy on the same levels, about half of the levels scored as learnable lose that score, at every stage of training.
- XLand-MiniGrid allows no statistical conclusion: one ruleset was never solved by any method and the other was solved by all.
What I took from it
The project was an exercise in running an empirical study end to end: fixing the comparisons before running them, keeping every number traceable to a run or a file, and following a negative headline result down to its cause. The most useful step was the ablation on solvable levels, which turned "the prior does not help" into a statement about when it does.
Code: preference annotation and difficulty-model training (uedrlhf) and SFL extensions (built on sampling-for-learnability). The thesis PDF is available on request. Language models were used for language refinement and coding assistance, as stated in the thesis.