These mini models are NOT the DeepSeek R1.<p>> DeepSeek-R1-Distill models are fine-tuned based on open-source models, using samples generated by DeepSeek-R1.<p>The amount of confusion on the internet because of this seems surprisingly high. DeepSeek R1 has 670B parameters, and it's not easy to run it on local hardware.<p>There are some ways to run it locally, like <a href="https://unsloth.ai/blog/deepseekr1-dynamic">https://unsloth.ai/blog/deepseekr1-dynamic</a> which should let you fit the dynamic quant into 160GBs of VRAM, but the quality will suffer.<p>Also MLX attempt on a cluster of Mac Ultras: <a href="https://x.com/awnihannun/status/1881412271236346233" rel="nofollow">https://x.com/awnihannun/status/1881412271236346233</a>
To run the full 671B Q8 model relatively cheaply (around $6k), you can get a dual EPYC server with 768GB RAM - CPU inference only at around 6-8 tokens/sec. <a href="https://x.com/carrigmat/status/1884244369907278106" rel="nofollow">https://x.com/carrigmat/status/1884244369907278106</a><p>There are a lot of low quant ways to run in less RAM, but the quality will be worse. Also, running a distill is not the same thing as running the larger model, so unless you have access to an 8xGPU server with lots of VRAM (>$50k), cpu inference is probably your best bet today.<p>If the new M4 Ultra Macs have 256GB unified RAM as expected, then you may still need to connect 3 of them together via Thunderbolt 5 in order to have enough RAM to run the Q8 model. Assuming that the speed of that will be faster than the EPYC server, but will need to test empirically once that machine is released.
> By default, this downloads the main DeepSeek R1 model (which is large). If you’re interested in a specific distilled variant (e.g., 1.5B, 7B, 14B), just specify its tag<p>No… it downloads the 7B model by default. If you think <i>that</i> is large, then you better hold on to your seat when you try to download the 671B model.
None of these models are the real Deepseek R1 that you can access via the API or chat!
The big one is a quantized version (it uses 4 bit per weight) and even that you probably cant run.<p>The other ones are fine-tunes of LLama 3.3 and Qwen2 which have been additionally trained on outputs of the big "Deepseek V3 + R1" model.<p>I'm happy people are looking into selfhosting models, but if you want to get an idea of what R1 can do, this is not a good way to do so.
Ollama is doing the community a serious disservice by presenting the various distillations of R1 as different versions of the same model. They're good improvements on their base models (on reasoning benchmarks, at least), but grouping them all under the same heading masks the underlying differences and contributions of the base models. I know they have further details on the page for each tag, but it still seems seriously misleading.
It's worth mentioning the fact that you need to adjust the context window manually for coding tasks (default 2k is not enough for coding tasks).<p>Here's how to run deepseek-r1:14b (DeepSeek-R1-Distill-Qwen-14B) and set it to 8k context window:<p><pre><code> ollama run deepseek-r1:14b
/set parameter num_ctx 8192
/save deepseek-r1:14b-8k
ollama serve</code></pre>
FYI, the 14b model fits and runs fine on a 12GB NVIDIA 3060, and is pretty capable, but still frustrating to use: <a href="https://news.ycombinator.com/item?id=42863228">https://news.ycombinator.com/item?id=42863228</a>
DeepSeek is available via HuggingFace and can be run in/from tools like LM Studio locally with enough Ram. 14B models is pretty impressive.<p>If you want the optionality of using the full model, there are private hosted models that can be connected in cheaply for those use cases into a "one place for all the models" locally.