I find some of these to be a little confusing, like an important detail is left out. In the RLHF one [0], for example, it makes sense that setting beta high pulls to distribution toward the "how likely the starting model was" distribution. But why does a low beta select for only the "Polished, wrong" and a low-ish beta (beta=0.16) allow for "Polished, wrong" and "Worked example", but completely suppress "I’m not sure" and "one line answer". It seems like there's some other distribution (the reward model maybe?) that's not shown in the interactive part, that makes it harder to understand than the text. It feels weird that the reward model part would be hidden and all the emphasis would be instead placed on the beta regularization parameter.
Commit history mentions Sonnet 5.5 so maybe that's the cause?