Training and aligning LLMs with RLHF and RLHF alternatives

Discussions about LLM alignment often miss topics of data quality and quantity. It turns out that current models like Llama 2 use 10K+ prompts and responses for supervised fine-tuning (SFT) and 100K+ human preference pairs. While the preferences are pretty easy to annotate, producing a good SFT dataset is uneasy.<p><a href="https:&#x2F;&#x2F;evalovernite.substack.com&#x2F;p&#x2F;rlhf-math-aint-enough" rel="nofollow noreferrer">https:&#x2F;&#x2F;evalovernite.substack.com&#x2F;p&#x2F;rlhf-math-aint-enough</a><p><a href="https:&#x2F;&#x2F;doi.org&#x2F;10.5281&#x2F;zenodo.8186168" rel="nofollow noreferrer">https:&#x2F;&#x2F;doi.org&#x2F;10.5281&#x2F;zenodo.8186168</a>

I read here that Yann LeCun claimed that even with RLHF, LLMs will still hallucinate - that it&#x27;s an unavoidable consequence of their autoregressive nature<p><a href="https:&#x2F;&#x2F;www.hopsworks.ai&#x2F;dictionary&#x2F;rlhf-reinforcement-learning-from-human-feedback" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.hopsworks.ai&#x2F;dictionary&#x2F;rlhf-reinforcement-learn...</a>