Alignment is not free: How model upgrades can silence your confidence signals

Very interesting! The one thing I don&#x27;t understand is how the author made the jump from &quot;we lost the confidence signal in the move to 4.1-mini&quot; and &quot;this is because of the alignment&#x2F;steerability improvements.&quot;<p>Previous OpenAI models were instruct-tuned or otherwise aligned, and the author even mentions that model distillation might be destroying the entropy signal. How did they pinpoint alignment as the cause?

there&#x27;s evidence that alignment also significantly reduces model creativity: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2406.05587" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2406.05587</a><p>it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas.

People use llm as part of their high precision systems? That’s worrying

Why not make a completely raw uncensored LLM? Seems it would be more &quot;intelligent&quot;.

Can we have models also return a probability, reflecting how accurate the statements it made is ?

It’s kinda ironic but parts of the article read like they were written by an LLLM itself

Upgrade scripts it is so. <i>plus ca change</i>

I don&#x27;t know if its still comedy or has now reached the stage of farce, but I still at least always get a good laugh when I see another article about the shock and surprise of researchers finding that training LLMs to be politically correct makes them dumber. How long until they figure out that the only solution is to know the correct answer but to give the politically correct answer (which is the strategy humans use) ?<p>Technically, why not implement alignment&#x2F;debiasing as a secondary filter with its own weights that are independent of the core model which is meant to model reality? I suspect it may be hard to get enough of the right kind of data to train this filter model, and most likely it would be best to have the identity of the user be in the objective.