Crazy amount of innovations in one technical report:<p>- successful fp8 quantized training for SOTA model<p>- multi token prediction, mostly to improve training results, but also to enable speculative decoding<p>- very high sparsity per request (37B activated params per 671B total params)<p>- using reasoning data (from DeepSeek R1) to fine-tune and improve results on math & coding<p>- manual balancing of compute / communication in their infrastructure, up to SM level
The big news here is the training costs, $5.576m total cost, equivalent to 2788k training hours on H800 GPU at $2 per hour. This for a model that is (according to DeepSeek's own benchmarks) SOTA for open source.