Accelerated PyTorch Training on M1 Mac

Curiously neither PyTorch nor Tensorflow currently use M1's Neural Engine. Is too limited? Too hard to interact with? Not worth the effort?

Since it&#x27;s tangentially relevant, if you have an M1 Mac I&#x27;ve created some boilerplate for working with the latest Tensorflow with GPU acceleration as well: <a href="https:&#x2F;&#x2F;github.com&#x2F;alexfromapex&#x2F;tensorexperiments" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;alexfromapex&#x2F;tensorexperiments</a> . I&#x27;m thinking of adding a branch for PyTorch now.

This is really cool for a number of reasons:<p>1.) Apple Silicon <i>currently</i> can&#x27;t compete with Nvidia GPUs in terms of raw compute power, but they&#x27;re already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now.<p>Edit: I&#x27;ve been informed that for matrix math, Apple Silicon isn&#x27;t actually ahead in efficiency<p>2.) Apple Silicon probably <i>will</i> compete directly with Nvidia GPUs in the near future in terms of raw compute power in future generations of products like the Mac Studio and Mac Pro, which is very exciting. Competition in this space is incredibly good for consumers.<p>3.) At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we&#x27;ll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking <i>very</i> enticing.

Nice results! But why are people still reporting benchmark results on VGG? Does anybody actually use this network anymore?<p>Better would be mobilenets or efficientNets or NFNets or vision transformers or almost anything that&#x27;s come out in the 8 years since VGG was published (great work it was at the time!).

The installation command generated on <a href="https:&#x2F;&#x2F;pytorch.org&#x2F;get-started&#x2F;locally&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pytorch.org&#x2F;get-started&#x2F;locally&#x2F;</a> didn&#x27;t install the latest version for me. What did it was:<p>pip3 install --pre torch==1.12.0.dev20220518 --extra-index-url <a href="https:&#x2F;&#x2F;download.pytorch.org&#x2F;whl&#x2F;nightly&#x2F;cpu" rel="nofollow">https:&#x2F;&#x2F;download.pytorch.org&#x2F;whl&#x2F;nightly&#x2F;cpu</a>

Exciting!! But don&#x27;t see comparison with any laptop Nvidia GPUs in terms of performance. That would be insightful.

This is very interesting since the M1 studio supports 128GB of unified memory - training a large memory heavy model slowly on a single device could be interesting, or inferencing a very large model.

There was a report comparing M1 Pro with several other Nvidia GPUs from a few months ago: <a href="https:&#x2F;&#x2F;wandb.ai&#x2F;tcapelle&#x2F;apple_m1_pro&#x2F;reports&#x2F;Deep-Learning-on-the-M1-Pro-with-Apple-Silicon---VmlldzoxMjQ0NjY3" rel="nofollow">https:&#x2F;&#x2F;wandb.ai&#x2F;tcapelle&#x2F;apple_m1_pro&#x2F;reports&#x2F;Deep-Learning...</a><p>I&#x27;m curious on how the benchmarks change with this recent new release!

Anyone actually got this to run on an M1 Mac?<p><pre><code> $ conda install pytorch torchvision torchaudio -c pytorch-nightly Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. PackagesNotFoundError: The following packages are not available from current channels: - torchaudio </code></pre> And the pip install variant installs an old version of torchaudio that is broken<p><pre><code> OSError: dlopen(&#x2F;opt&#x2F;homebrew&#x2F;Caskroom&#x2F;miniforge&#x2F;base&#x2F;envs&#x2F;test123&#x2F;lib&#x2F;python3.10&#x2F;site-packages&#x2F;torchaudio&#x2F;lib&#x2F;libtorchaudio.so, 0x0006): Symbol not found: __ZN2at14RecordFunctionC1ENS_11RecordScopeEb</code></pre>

I&#x27;m curious about the performance compared to something like, say, the RTX 3070.

Small code example in the PyTorch doc:<p><a href="https:&#x2F;&#x2F;pytorch.org&#x2F;docs&#x2F;master&#x2F;notes&#x2F;mps.html" rel="nofollow">https:&#x2F;&#x2F;pytorch.org&#x2F;docs&#x2F;master&#x2F;notes&#x2F;mps.html</a>

Anyone else getting &quot;illegal hardware instruction&quot;?<p>(pytorch_env) ~&#x2F;dev&#x2F;ai&#x2F; python -c &quot;import torch&quot;

It’s surprising to see PyTorch developers working on things like that when common operations like group convolutions are still completely unoptimized on Nvidia GPUs, despite many requests.

This is much nicer ergonomics than what I had to do for tensorflow. It’s ostensibly out of the box support as a different torch device.

I started collecting benchmarks of the M1 Max on PyTorch here: <a href="https:&#x2F;&#x2F;github.com&#x2F;lucadiliello&#x2F;pytorch-apple-silicon-benchmarks" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;lucadiliello&#x2F;pytorch-apple-silicon-benchm...</a>

yess! This is important for me, because I don&#x27;t have any $$$ to rent GPUs for personal projects. Now we just need M1 support for JAX.<p>Since there are no hard benchmarks against other GPUs, here&#x27;s a Geekbench against an RTX 3080 Mobile laptop I have [1]. Looks like it&#x27;s about 2x slower--the RTX laptop absolutely rips for gaming, I love it.<p>[1] <a href="https:&#x2F;&#x2F;browser.geekbench.com&#x2F;v5&#x2F;compute&#x2F;compare&#x2F;4140651?baseline=4529092" rel="nofollow">https:&#x2F;&#x2F;browser.geekbench.com&#x2F;v5&#x2F;compute&#x2F;compare&#x2F;4140651?bas...</a>

A tangential thought: will we see animation studios buy mac studios for their rendering farms? What do they use these days, aws ec2?

Anyone care to comment on how this is better than Metal&#x27;s TensorFlow support?

Does this work on any Metal hardware or just the M1 GPU?

really hope to see some comparisons with nvidia gpus!

Does speed up refer to absolute value or percentage?

What is the * in the chart referencing?

&gt; Accelerated GPU training is enabled using Apple’s Metal Performance Shaders (MPS) as a backend for PyTorch.<p>What do shaders have to do with it? Deep learning is a mature field now, it shouldn&#x27;t need to borrow compute architecture from the gaming&#x2F;entertainment field. Anyone else find this disconcerting?