Optimizing AI Inference at Character.ai

Noam has been cooking at character.ai in stealth. Their model is impressively engaging

Training in int8 is noteable (to me). I've been out of date with ML research for a bit now but last I recall, people were mostly training at full precision and then quantizing after training and finetuning a bit on the quantized model afterwards.

&gt; we implemented customized int8 kernels for matrix multiplications and attention<p>I would be curious how this differs from [1] which is supported in Huggingface’s transformers library.<p>[1] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2208.07339" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2208.07339</a>