DeepSeek-V3 Technical Report

The GPU-hours stat here allows us to back out some interesting figures around electricity usage and carbon emissions if we make a few assumptions.<p>2,788,000 GPU-hours * 350W TDP of H800 = 975,800,000 GPU Watt-hours<p>975,800,000 GPU Wh * (1.2 to account for non-GPU hardware) * (1.3 PUE [1]) = 1,522,248,000 Total Wh, or 1,522,248 kWh to train DeepSeek-V3<p>(1,522,248 kWh) * (0.582kg CO2eq&#x2F;kWh in China [2]) = 885,948 kg CO2 equivalents to train DeepSeek-V3<p>A typical US passenger vehicle emits about 4.6 metric tons of CO2 per year. [3]<p>885,948 kg CO2 per DeepSeek &#x2F; 4,600 kg CO2 per car = 192.6 cars per DeepSeek<p>So, the final training run for DeepSeek-V3 emitted as much greenhouse gasses as would be emitted from running about 193 more cars on the road for a year.<p>I also did some more math and found that this training run used about as much electricity as 141 US households would use over the course of a year. [4]<p>[1] <a href="https:&#x2F;&#x2F;enviliance.com&#x2F;regions&#x2F;east-asia&#x2F;cn&#x2F;report_10060" rel="nofollow">https:&#x2F;&#x2F;enviliance.com&#x2F;regions&#x2F;east-asia&#x2F;cn&#x2F;report_10060</a><p>[2] <a href="https:&#x2F;&#x2F;ourworldindata.org&#x2F;grapher&#x2F;carbon-intensity-electricity" rel="nofollow">https:&#x2F;&#x2F;ourworldindata.org&#x2F;grapher&#x2F;carbon-intensity-electric...</a><p>[3] <a href="https:&#x2F;&#x2F;www.epa.gov&#x2F;greenvehicles&#x2F;greenhouse-gas-emissions-typical-passenger-vehicle" rel="nofollow">https:&#x2F;&#x2F;www.epa.gov&#x2F;greenvehicles&#x2F;greenhouse-gas-emissions-t...</a><p>[4] divided total kWh by the value here: <a href="https:&#x2F;&#x2F;www.eia.gov&#x2F;tools&#x2F;faqs&#x2F;faq.php?id=97&amp;t=3" rel="nofollow">https:&#x2F;&#x2F;www.eia.gov&#x2F;tools&#x2F;faqs&#x2F;faq.php?id=97&amp;t=3</a>

The fact that you can unironically put the &quot;only&quot; modifier on a training time of 2.8 million GPU hours is nuts.

Re DeepSeek-V3 0324 - I made some 2.7bit dynamic quants (230GB in size) for those interested in running them locally via llama.cpp! Tutorial on getting and running them: <a href="https:&#x2F;&#x2F;docs.unsloth.ai&#x2F;basics&#x2F;tutorial-how-to-run-deepseek-v3-0324-locally">https:&#x2F;&#x2F;docs.unsloth.ai&#x2F;basics&#x2F;tutorial-how-to-run-deepseek-...</a>

Hasn&#x27;t been updated for the -0324 release unfortunately, and diff-pdf shows only a few small additions (and consequent layout shift) for the updated arxiv version on Feb 18.

Nice to see a return to open source in models and training systems.

<a href="https:&#x2F;&#x2F;x.com&#x2F;iScienceLuvr&#x2F;status&#x2F;1905144432791609480" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;iScienceLuvr&#x2F;status&#x2F;1905144432791609480</a>

I like that they give advice to hardware manufacturers: - offload communication to a dedicated co-proc - implement decent precision for accumulating fp8 operations - finer-grained quantization ...