I thought to myself this morning: "boy, that $15k pricetage is tempting." Then I thought to myself "how many times have I downloaded a github repo only to hand-replace cuda with mps, and then tried to figure out if there's a version of xformers that will work this week with my m3?" and then I thought "boy, that $25k is tempting." (15k: Radeon / 25k: Nvidia).<p>For those wondering, 3200W power, in residential / low-end commercial in the US, they say you'll need two separate circuits, they have a built-in power limiting utility in the OS which will let you safely run on one circuit at reduced speed.<p>The only part of this that gives me pause is interconnect -- over PCIe, 64GB/s stated. This is much, much lower than infiniband -- can any ML engineers comment on using this box for a full finetune of, say, LLama 3.1 / 70b?
I had a preorder in for this but I canceled a few weeks back.<p>My experience trying to run machines this powerful in residential settings has been extremely poor.<p>All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time.<p>Even after violating local electrical code, I have found that living with a 1500w+ monster inside my house <i>during the summer</i> at 100% utilization is a complete joke. Unless you live in the perfect datacenter climate (i.e. the people who designed the tiny box), this thing needs to be inside. All of that wattage is pure heat being dumped into your home. The HVAC solutions in most residences were not designed for this kind of heat load. It would be like running your oven with the door hanging open all day. For those of us in places like Texas, this machine simply would not be feasible to run for half the year.
<a href="https://tinygrad.org/#tinybox" rel="nofollow">https://tinygrad.org/#tinybox</a><p>Looks like good value, but I wonder if it would get CPU/RAM bottlenecked, especially if you want to train something with a lot of preprocessing in the pipeline. Something comparable I've found with 7x4090 which comes to about $50k, but with much better CPU/RAM (3x CPU, 4x RAM, 5x SSD):<p><a href="https://www.overclockers.co.uk/8pack-supernova-mk3-amd-ryzen-threadripper-pro-extreme-pc-sys-8pk-00076.html" rel="nofollow">https://www.overclockers.co.uk/8pack-supernova-mk3-amd-ryzen...</a>
I looked at the specs at the start of the year and just built something with the high end of consumer parts at around 4k usd. I was able to replicate the mlc 2x7900xtx results running some LLMs. Good enough to run most of the big models in gpu memory with a little quantization.
So is the plan for these to quietly update the hardware as better consumer hardware becomes available? This is a really interesting idea but as a small fry I would definitely be building myself if I went this route.
Very unusual specs on paper.<p>- Air cooling 6x4090 and a 32 core CPU for sustained peak workloads.<p>- 3200W total power when a single 4090 can draw close to 600W.<p>Maybe they are targeting startups who aren't interested in overclocking.
Unless you're making money off it, $15k + however much you have to spend on installing a new breaker panel is too much to spend on hardware that will be outdated in 2 years. If you're making money off it, but you're still cheap, then buy a Supermicro + H100s and colo it in a datacenter. If you're not cheap, you'll just use Azure. So I'm not sure who this product is supposed to be for.
> The $15k tinybox red is the best perf/$ ML box in the world. It's fully networkable, <i>so that's the metric that matters</i>.<p>No it isn't. Capex is only part of the equation. Opex (power and cooling amongst other things) is important. And networking at scale isn't cheap either.
Anyone know how well this would compare to the Nvidia based workstations at the GPTShop.ai place?<p>ie: things like this <a href="https://gptshop.ai/config/indexus.html" rel="nofollow">https://gptshop.ai/config/indexus.html</a>