Appendix D — Reinforcement Learning for Recommendation

The bandit chapter let the recommender choose what to show and learn from the feedback, but under one quiet assumption: each action is one-shot, so what you show now does not change what situation you face next. Reality often breaks that. Recommend episode 1 and episode 2 becomes relevant that wasn’t before; fill a feed with clickbait and you win today’s clicks but lose tomorrow’s session. The moment actions move the world forward, you are no longer maximizing the next click; you are maximizing long-term reward, and that is reinforcement learning (RL).

What this appendix assumes. It assumes the bandit setting of Bandits & Online Recommendation (the bandit is the one-step special case of what follows) and the idea of a click reward. Sections 1-4 also lean on the expected value and standard deviation of Probability & Statistics for Machine Learning and on gradient ascent (gradient descent of Calculus for Machine Learning with the sign flipped, to climb instead of descend); section 4 adds normalized discounted cumulative gain (NDCG; Evaluation Metrics), cross-entropy (Loss Functions & Regularizers §2.2) and Large Language Models and Recommendation §2.2. It is a thin frontier appendix: a map, not a full course, with hand-checkable micro-examples, four of them the main ones (a discounted return and a choice that long-term reward changes, both in §1; a slate’s value, in §2; a group-relative advantage, in §4); for the full theory, Sutton & Barto is the text.

1. From bandits to reinforcement learning

Suppose a streaming service picks what to show Ann tonight. If the pick changes what she will want tomorrow, judging each pick by its own click is not enough, and we need a way to write the whole situation down. RL writes it as a Markov Decision Process (MDP) (Markov: the state carries everything about the past that matters for what comes next), which has four pieces and a discount:

  • a state \(s\) — a summary of the situation now (the user’s recent history, time, context);
  • an action \(a\) — what the agent (the recommender) does (which item, or slate (a page of items), to show);
  • a reward \(r\) — the immediate payoff (a click, a watch-minute, a return visit);
  • a transition — taking action \(a\) in state \(s\) moves you to a new state \(s'\), possibly random (drawn from a distribution), and in an MDP actions can change the next state;
  • a discount factor \(\gamma\) (below): how much a later reward is worth.

For example, Ann’s state is “watched 3 comedies this week”, the action is to show a comedy slate, the reward is \(1\) (she clicks), and the transition takes her to “watched 4 comedies” with probability \(0.8\) or to “left the app” with probability \(0.2\).

A policy \(\pi(a\mid s)\) is not one of the pieces: it is the solution, the rule mapping states to actions, which is what we learn. Here \(\pi\) names the policy, not \(3.14159\), and \(\pi(a\mid s)\) is the probability that it picks action \(a\) in state \(s\) (in the choice example below, always showing quality is \(\pi(\text{quality}\mid s)=1\)).

The goal is not the next reward but the return \(G\). We maximize its expected value, averaged over the random transitions, rewards and the policy’s own choices. The return is the cumulative discounted long-term reward, \(G = r_0 + \gamma\,r_1 + \gamma^2\,r_2 + \cdots = \sum_{t\ge 0}\gamma^{t}\,r_t\), where \(\gamma\in[0,1)\) is the discount factor — how much less a reward one step later is worth than one now (it both encodes “sooner is better” and, for bounded rewards, keeps an infinite sum finite: a reward of \(1\) at every step sums to \(1/(1-\gamma)=10\) at \(\gamma=0.9\)).

Worked example — a discounted return. Suppose over three steps the agent collects rewards \(r = (1,\,0,\,1)\) with discount \(\gamma = 0.9\). Then \(G = 1 + 0.9(0) + 0.9^2(1) = 1 + 0 + 0.81 = 1.81.\) The click two steps out is worth only \(\gamma^2 = 0.81\) of an immediate one — the same click, discounted for being later. Push \(\gamma\to 0\) and the objective collapses to a bandit-like myopic “maximize the immediate reward”; push \(\gamma\to 1\) and the far future matters almost as much as now: a reward ten steps out keeps \(0.9^{10}=0.349\) of its worth at \(\gamma=0.9\) but \(0.99^{10}=0.904\) at \(\gamma=0.99\). (The figure below draws the three steps.)

The discounted return G=sumtgamma^t rt on the §1 worked example (r=(1,0,1), gamma=0.9).
Figure D.1: The discounted return \(G=\sum_t\gamma^t r_t\) on the §1 worked example (\(r=(1,0,1)\), \(\gamma=0.9\)). Each bar is one step’s discounted reward \(\gamma^t r_t\): the immediate \(r_0\) counts in full, but the identical reward at \(t{=}2\) is shrunk to \(\gamma^2=0.81\). The bars sum to \(G=1.81\). Send \(\gamma\to0\) and only the first bar survives (a myopic, bandit-like objective); send \(\gamma\to1\) and every bar counts equally.

The value \(V(s)\) is the expected return from state \(s\) under a policy, and the action-value \(Q(s,a)\) the expected return from taking \(a\) in \(s\) and then following that policy (the box below prints both: \(Q(\text{quality})=1.4\) and the value \(V(s)=1.4\) of the greedy policy). RL methods learn one of these and act greedily (take the action with the highest \(Q\)) or explore, as a bandit does; policy-gradient methods (§3) instead learn the policy directly.

Q-learning (Watkins & Dayan, 1992) learns \(Q(s,a)\) by moving each estimate toward the reward now plus \(\gamma\) times the best \(Q\) at the next state (the idea behind temporal-difference (TD) learning, Sutton, 1988). For example, with a current estimate \(Q=1.0\), reward \(0.5\), best next-state \(Q=1.4\) and \(\gamma=0.9\), the target is \(0.5+0.9\cdot1.4=1.76\), and a step size (the learning rate) of one half moves the estimate to \(1.0+0.5(1.76-1.0)=1.38\).

Worked example — long-term reward changes the choice. A user is in one state with two actions. Clickbait pays \(r=1\) now and the user then leaves (\(0\) next); quality pays \(0.5\) now and the user stays for a reward of \(1\) next (these later rewards are given, not modeled further). With \(\gamma=0.9\), \(Q(\text{clickbait})=1+0.9(0)=1.0\) and \(Q(\text{quality})=0.5+0.9(1)=1.4\), so the greedy policy shows quality and its value is \(V(s)=\max_a Q(s,a)=1.4\) (a policy that shows each action half the time has \(V=\tfrac12\cdot1.0+\tfrac12\cdot1.4=1.2\): \(V\) depends on the policy). A bandit, which sees only the first reward, picks clickbait (\(1>0.5\)); at \(\gamma=0\) the two are \(1.0\) and \(0.5\), exactly the bandit’s choice. Quality wins whenever \(0.5+\gamma>1\), that is \(\gamma>0.5\).

A bandit is an MDP in which actions never change the next state: one state (the plain bandit) or a fresh context each round (the contextual bandit). That is why the explore / exploit machinery of Bandits & Online Recommendation reappears here, now wrapped around long-term value. The figure below draws the loop.

The reinforcement-learning loop. The agent picks an action from its policy; the environment (the user) returns a reward and a new state.
Figure D.2: The reinforcement-learning loop. The agent picks an action from its policy; the environment (the user) returns a reward and a new state. What bandits lack is an action that can change the new state; because it can, the agent must optimize the discounted return \(G=\sum_t\gamma^t r_t\), not just the next reward. A bandit is this loop in which the action never changes the next state.

In short: RL is a bandit whose actions also move the state, judged by the discounted return instead of the next reward alone.

2. Why RL is genuinely hard in recommendation

RL is alluring for recommendation — “optimize long-term engagement, not next-click clickbait” — but four obstacles make it hard in practice, and a from-zero reader should know them before reaching for it. The four obstacles are claims from practice, not tested here (the decomposition in the first is the cited exception):

  1. A combinatorial action space. The action is often a whole slate of items, so the number of actions is astronomically large. Q-learning (§1) must score every action: a page of 10 items from 1,000 candidates is about \(10^{30}\) ordered slates, so plain Q-learning cannot be used. SlateQ (slate Q-learning; Ie et al., 2019) is one fix: if the user consumes one item per slate (or none) and reward and transition depend only on it, a slate’s value decomposes into per-item \(Q\)-values weighted by the chance each item is chosen. For per-item values \(2,1,0.5\) chosen with probabilities \(0.4,0.3,0.2\) (from an assumed choice model, the user’s rule for picking among the shown items; the other \(0.1\) is no choice, taken as worth \(0\)) the slate is worth \(0.4\cdot2+0.3\cdot1+0.2\cdot0.5=1.2\). Choosing the best slate is still a separate optimization (an exact linear program, or a greedy rule).
  2. It is off-policy in practice. You cannot let a half-trained agent experiment on real users for months, so you must learn from logged data collected by a different (old) policy — the hard, bias-prone off-policy setting (the same logged-data problem behind the bandit offline replay of Bandits & Online Recommendation). For example, a log of 1,000 sessions in which the old policy showed comedy \(90\%\) of the time gives only about 100 sessions of anything else, too few to say what a new policy that shows documentaries would get.
  3. Delayed, sparse reward + the simulator gap. “Long-term satisfaction” arrives much later than the action that caused it and rarely (credit assignment, deciding which earlier action deserves the credit, is hard: if Ann subscribes on day 30 after 30 recommendations, splitting the credit equally gives each \(1/30\), which is only a guess), and RL often needs very many interactions, so teams build user simulators, which are themselves recommendation models and can bake in their own errors (a simulator that says Ann clicks comedy with probability \(0.7\) is only as right as that \(0.7\)).
  4. Safety and non-stationarity (the data’s pattern changing over time). An exploring policy on a live platform can hurt real users and revenue, and user behavior drifts under the policy itself.

So a slate rules out plain Q-learning, and the other three obstacles are about data, reward and safety, which the choice of algorithm only partly fixes.

3. The main approaches, in one line each

Once the obstacles are in view, the question is what to learn: how good each action is, or directly what to do. The third item below is a deep-network version of the first.

  • Value-based. Learn \(Q(s,a)\) (Q-learning) and act greedily; for slates, SlateQ makes it tractable by decomposition.
  • Policy-gradient. Directly optimize the policy \(\pi_\theta\) (the policy with parameters \(\theta\)) by gradient ascent on expected return. The basic method, REINFORCE (Williams, 1992), raises the probability of each action taken in proportion to the return that followed: with no baseline (a reference return, often the average, subtracted from each return) and positive returns, a title shown with return \(2.0\) is pushed up twice as hard as one with return \(1.0\). Chen et al. (2019) use an off-policy-corrected form (§2, item 2) at web scale, for action sets of millions of items.
  • Deep RL for news (DRN). Zheng et al. (2018) use deep Q-learning (Mnih et al., 2015: the \(Q\) values come from a neural network) for online news, explicitly modeling future reward and how soon the user comes back, a signal beyond the immediate click, a concrete case of optimizing for retention rather than the next tap.

The approaches differ in what they must learn: values (plus a choice model for slates) or the policy itself, with a neural network standing in for the values in the third.

4. When the policy is a language model

Ask a chat model what Ann should watch tonight and it simply writes an answer; nothing yet says whether the answer was good. This section is about the scoring that turns such an answer into a learning signal.

A recommender built on a large language model (LLM) reuses §1’s words with one change of scale. The state \(s\) is the prompt (the user’s history, perhaps a query). The action \(a\) is the text the model writes: one item title, a ranked list or an explanation. The policy \(\pi_\theta(a\mid s)\) is the probability the model gives that text, piece by piece (each piece, a token, is a word or part of one). The reward scores the finished output. In the setups below the model writes its output in one go, so there is no next state and no discounting over time: this is nearer to the contextual bandit of §1 than to a long-horizon MDP.

Some methods below skip the reward in their final step and learn only from pairs of a preferred and a rejected output (for example, the title a user chose and one shown but not chosen). For the models themselves, see Large Language Models and Recommendation §2.2.

A learned reward. Reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Ouyang et al., 2022) trains a reward model (a model that scores outputs) on preferred and rejected outputs, then tunes the model against it while a penalty keeps it near its starting point; direct preference optimization (DPO; Rafailov et al., 2023, cited in Large Language Models and Recommendation §2.2) merges both steps into one loss on pairs. Softmax-DPO (S-DPO; Chen et al., 2024) widens the pair to one preferred item against several negatives (items the user did not choose), and its ablation (a test that removes one part to see what it was worth), which keeps one negative so that the loss reduces to plain DPO, reports gains from the extra negatives.

OneRec (Deng et al., 2025, a generative recommender from the video platform Kuaishou) starts from a problem: a platform shows one list per request, so it never sees a winner and a loser side by side. Its authors therefore train a reward model on whole sessions, with a binary cross-entropy loss (the yes/no log-loss of Loss Functions & Regularizers §2.2) against the logged outcomes of each session (watch time, views, follows and likes), and use it to rank the model’s own output: 128 lists per user from beam search (a search that keeps only the best few partial lists at each step), the highest-reward list paired with the lowest-reward one, so that the model builds for itself the winner-and-loser pair the logs never show. They apply this to a random 1% of the data, to limit the cost of the beam search.

A rule as the reward. Rec-R1 (Lin et al., 2025) scores text the model writes by the NDCG (a ranking-quality score) of a fixed recommender’s output (for example, the model rewrites Ann’s search query, the fixed recommender returns a ranked list for the rewrite, and that list’s NDCG is the reward), uses group relative policy optimization (GRPO; Shao et al., 2024: score a group of outputs against their own mean, as in the example below) and trains no reward model, which its authors say limits reward hacking (the model learns to raise the score without improving the recommendations).

ReRe (Reinforced Preference Optimization for Recommendation; Tan et al., 2025) gives reward \(1\) when a generated title equals the target and \(0\) otherwise, and adds a ranking reward for the all-zero case (see the example). Rank-GRPO (Zhu et al., 2025), written for conversational recommenders that output ranked lists, adapts GRPO by treating each rank, rather than each token or the whole sequence, as the unit that is optimized and crediting each rank with the discounted relevance of itself and the ranks below it; its authors found that crediting only the rank’s own item (no look-ahead, so \(\gamma=0\) in §1’s terms) gave a simple, stable learning signal.

Worked example (GRPO). The model writes \(n=4\) titles for one prompt (the group size) and one matches the target, so the rewards are \((1,0,0,0)\). Their mean is \(0.25\) and their sample standard deviation \(0.5\), so the advantages \((r-\text{mean})/\text{std}\) are \((1.5,-0.5,-0.5,-0.5)\): the hit is made likelier, the misses less likely, and no value model (a second network that estimates \(V\)) is trained. If no title hits, every reward is \(0\) and the group gives no signal, the all-zero case for which ReRe adds a ranking reward.

What is open. ReRe reports that smoother rewards (text similarity, another recommender’s scores) kept rising while accuracy fell. In the parts read, none of these papers optimizes a multi-step return over time, such as retention, so §2’s delayed-reward problem is untouched. In the parts read, only OneRec reports an online test, for a whole system (Large Language Models and Recommendation §2.2); ReRe, OneRec and Rank-GRPO are cited here as preprints, and the papers’ datasets differ, so their gains are not comparable. All of this is reported there, not tested here. The takeaway is that with a language model as the policy, the new question is where the reward comes from (a reward model, pairs, or a rule), not the long-horizon return.

5. Where it pays off — and an honest caveat

RL earns its complexity when the objective is genuinely long-horizon: total session length, multi-step journeys (a course syllabus, a show’s episode order), or retention (will the user come back next week?) — outcomes a one-step click model cannot express and a bandit can only credit to a single action. The caveat, in the spirit of the reproducibility caution of Datasets & Benchmarks §5: RL in recommendation is operationally heavy and often hard to reproduce, and a well-tuned bandit or supervised ranker is often competitive on the metrics that are actually measured. This is a view from practice, not tested here (the cited section shows the same caution for neural recommenders against tuned classical baselines, not for RL). Reach for RL when the objective itself is long-term and you can pay the engineering cost, not by default.

What the reading behind this appendix covers. The author’s reading of Chen et al. (2019), DRN and Afsar et al. rests on abstracts and scope. The reading of the §4 papers rests on abstracts and methods (for S-DPO also the ablation, for Rank-GRPO the reward definitions), and the reported results were not reproduced.

This is why RL lives here as a pointer appendix rather than a core chapter: the book’s through-line is collaborative filtering from classical to graph- and LLM-based models, and RL is an adjacent paradigm the through-line touches (via bandits) but does not need. For the one-step case, return to Bandits & Online Recommendation; for the full theory and a survey of the field (Afsar et al., 2022), the references below.

6. Glossary

Term Plain meaning
MDP Markov Decision Process: states, actions, transition, reward and discount — the problem RL solves; the policy is its solution.
Reinforcement learning (RL) Learning to choose actions to maximize long-term reward when actions can change the next state.
Reward \(r\) / action \(a\) The immediate payoff (a click, a watch-minute, a return visit) / what the recommender does (show an item or slate).
State \(s\) A summary of the current situation; what the agent conditions on.
Bandit An MDP in which actions never change the next state (one state, or a fresh context each round); such actions are called one-shot.
Policy \(\pi(a\mid s)\) The learned rule mapping a state to an action (\(\pi\) names the policy, not \(3.14159\)); what RL optimizes. A language model’s output rule in reinforcement learning from human feedback (RLHF) is also called a policy (Probability & Statistics for Machine Learning, glossary); §4 applies the idea to LLM-based recommenders.
GRPO Group relative policy optimization: sample a group of outputs for one prompt and score each against the group’s mean and spread, with no value model (§4).
Reward model A model trained to score outputs: RLHF trains it on preferred and rejected outputs, OneRec on the logged watch-time, view, follow and like outcomes of whole sessions; Rec-R1 uses none (§4).
Return \(G\) Cumulative discounted reward \(\sum_t \gamma^t r_t\) — the long-term objective.
Discount \(\gamma\) How much a later reward is worth vs. now; \(\gamma\to0\) is myopic (cares only about the immediate reward, like a bandit’s objective), \(\gamma\to1\) far-sighted.
Value / \(Q\)-value Expected return from a state (\(V\)) or a state–action pair (\(Q\)).
Q-learning Learn \(Q(s,a)\) by moving it toward reward plus \(\gamma\times\) the best next \(Q\) (temporal-difference learning), then take the highest-\(Q\) (greedy) action; it must score every action, so slate methods avoid it. Deep Q-learning uses a neural network for \(Q\); DRN applies it to news.
Policy gradient Optimize the policy directly by gradient ascent on expected return (REINFORCE is the basic algorithm; \(\theta\) are the policy’s parameters).
Retention Whether the user comes back (next week); a long-horizon objective.
Credit assignment Deciding which earlier action deserves credit for a reward that arrives later.
User simulator A model of user behavior used to train RL without experimenting on real users.
Direct preference optimization (DPO) / S-DPO A loss that learns from pairs of a preferred and a rejected output with no separate reward model; Softmax-DPO (S-DPO) widens the pair to one preferred item against several negatives (§4).
Advantage How much better an output scored than the group’s mean reward, divided by the group’s standard deviation in GRPO (§4).
Ablation A test that removes one part of a method to see what the part was worth (§4).
Beam search A search that keeps only the best few partial lists at each step (§4).
Slate A whole page/list of recommended items — the (combinatorial) action in slate RL.
Off-policy Learning from data collected by a different (older) policy than the one being trained.
SlateQ Decomposes a slate’s value into per-item \(Q\)-values, making value-based slate RL tractable.

7. References

Entries are grouped by topic (reinforcement-learning foundations, recommender systems, language-model methods), not by the alphabet.

  • Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press. (the standard text: MDPs, temporal-difference (TD) learning, Q-learning, policy gradients).

  • Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9-44. https://doi.org/10.1007/BF00115009 (temporal-difference learning).

  • Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3-4), 279-292. https://doi.org/10.1007/BF00992698 (the Q-learning update).

  • Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4), 229-256. https://doi.org/10.1007/BF00992696 (introduces REINFORCE).

  • Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236 (deep Q-learning).

  • Ie, E., Jain, V., Wang, J., Narvekar, S., Agarwal, R., Wu, R., Cheng, H.-T., Chandra, T., & Boutilier, C. (2019). SlateQ: A tractable decomposition for reinforcement learning with recommendation sets. In Proceedings of IJCAI. https://doi.org/10.24963/ijcai.2019/360

  • Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., & Chi, E. H. (2019). Top-K off-policy correction for a REINFORCE recommender system. In Proceedings of WSDM. https://doi.org/10.1145/3289600.3290999

  • Zheng, G., Zhang, F., Zheng, Z., Xiang, Y., Yuan, N. J., Xie, X., & Li, Z. (2018). DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of WWW. https://doi.org/10.1145/3178876.3185994

  • Afsar, M. M., Crump, T., & Far, B. (2022). Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7), 1-38. https://doi.org/10.1145/3543846

  • Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30 (NeurIPS ’17). https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html arXiv:1706.03741

  • Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS ’22) (pp. 27730-27744). https://doi.org/10.52202/068431-2011 arXiv:2203.02155 (RLHF applied to a language model).

  • Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 (preprint; introduces GRPO).

  • Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36 (NeurIPS ’23) (pp. 53728-53741). https://doi.org/10.52202/075280-2338 arXiv:2305.18290

  • Chen, Y., Tan, J., Zhang, A., Yang, Z., Sheng, L., Zhang, E., Wang, X., & Chua, T.-S. (2024). On softmax direct preference optimization for recommendation. In Advances in Neural Information Processing Systems 37 (NeurIPS ’24) (pp. 27463-27489). https://doi.org/10.52202/079017-0863 arXiv:2406.09215

  • Deng, J., Wang, S., Cai, K., Ren, L., Hu, Q., Ding, W., Luo, Q., & Zhou, G. (2025). OneRec: Unifying retrieve and rank with generative recommender and iterative preference alignment. arXiv:2502.18965 (preprint; see Large Language Models and Recommendation for its reported deployment).

  • Lin, J., Wang, T., & Qian, K. (2025). Rec-R1: Bridging generative large language models and user-centric recommendation systems via reinforcement learning. Transactions on Machine Learning Research. arXiv:2503.24289

  • Tan, J., Chen, Y., Zhang, A., Jiang, J., Liu, B., Xu, Z., Zhu, H., Xu, J., Zheng, B., & Wang, X. (2025). Reinforced preference optimization for recommendation. arXiv:2510.12211 (preprint; no venue was listed on its arXiv page).

  • Zhu, Y., Steck, H., Liang, D., He, Y., Ostuni, V., Li, J., & Kallus, N. (2025). Rank-GRPO: Training LLM-based conversational recommender systems with reinforcement learning. arXiv:2510.20150 (preprint; its latest arXiv PDF carries an ICLR 2026 header, but its arXiv page listed no venue).