[janalsncm]

I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.

[porridgeraisin]

GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.

[janalsncm]

It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.

[porridgeraisin]

Yes, but monte carlo estimating the critic model is not new.