ARM Mac: Why I'm Worried About Virtualization

I&#x27;m actually not worried, for a few reasons;<p>- I already do cross-arch development day-in and day out between x86 and ARM, and have only run into hard blockers on a library or tool a handful of times. The solve was generally pretty straightforward to either use an ARM-compatible alternative, or to cross-compile it myself.<p>- We&#x27;ve done this many many times before and it&#x27;s not that bad. I know I&#x27;m not the only one old enough here to remember the days of having heterogeneous fleets across PPC, SPARC, and x86. Or even more recent - different extensions for x86 with different chipset manufacturers.

It is very likely that ARM-based Macs will lack a performant hypervisor upon release. We will have to see how VMWare responds. I&#x27;d bet it will inspire new products and innovation and the desktop space will move towards a less x86-x64 centric world. In the end it is a short term problem. Someone will respond and provide a performant hypervisor that can run on an ARM host and virtualize x86-x64 and ARM guests.<p>It&#x27;s true it will cause some pain in the first year or two, but even as a heavy VMWare Fusion user I am really looking forward to the benefits of a vertically integrated laptop.

I&#x27;d say that I&#x27;m excited for ARM. That doesn&#x27;t mean the transition will be seamless or easy.<p>I know that a big complaint about the move is &quot;great, now I&#x27;m doing ARM locally and deploying to x86&quot;. I think this is a legitimate concern, for now, but I also strongly believe it is inevitable that, within the next decade, deploying to x86 in the Cloud will be as &quot;weird&quot; as ARM would be today. The benefits are way too numerous.<p>Well, more accurately, I think it&#x27;ll be a &quot;I&#x27;m on Fargate, oh wow, Fargate runs on ARM, I had no idea&quot; kind of thing. Ok, the article outlines why you may need <i>some</i> idea, but come on; we&#x27;re talking about one line where I&#x27;m downloading the x86 version of a dependency instead of an ARM version. That&#x27;s an easy fix.<p>I don&#x27;t know what this means for open accessibility of hardware. Right now, I could go buy and run locally the Intel Xeon chip powering my app in the cloud; when things move to ARM, it absolutely will be &quot;AWS Graviton&quot; (not sold outside AWS) or &quot;Azure ARM Whatever&quot; (not sold outside Azure). This sucks for accessibility, but, actually, does it? ARM enables the cloud providers to do this; they could never design their own x86 chips. As long as we&#x27;re all standardized on the same ISA, and the chips generally have the same characteristics, I&#x27;m looking forward to a very bright future where vendors are now also competing against one-another <i>in the silicon</i>. And I may not be able to buy an AWS Graviton, but I&#x27;m sure (well, hopeful) that one day I&#x27;ll be able to build an ARM desktop that isn&#x27;t a Raspberry Pi. AWS will have their chips, Quallcomm has theirs, Apple has theirs, Microsoft and Google have some, and they&#x27;re all competing against one another.<p>Ok, maybe this is a pipe dream. But, I&#x27;m definitely in the short Intel camp, at least on the long-term.

This seems like a weird benchmark, reading from &#x2F;dev&#x2F;urandom and gzipping random data does not seem like something most folks will want to do. It even appears like &#x2F;dev&#x2F;urandom speeds differ greatly on various architectures [0] and there are issues with &#x2F;dev&#x2F;random being fundamentally slow due to the entropy pool [1] (but I guess this is why the author uses &#x2F;dev&#x2F;urandom).<p>It would be better to measure something more related to what docker users will actually do, like container build time of a common container, and&#x2F;or latency of HTTP requests to native&#x2F;emulated containers running on the some container.<p>One reason to feel positive about the virtualization issues is that Rosetta 2 provides x86-&gt;ARM translation for JITs which an ARM-based QEMU could perhaps integrate into it&#x27;s own binary translation [2].<p>[0] <a href="https:&#x2F;&#x2F;ianix.com&#x2F;pub&#x2F;comparing-dev-random-speed-linux-bsd.html" rel="nofollow">https:&#x2F;&#x2F;ianix.com&#x2F;pub&#x2F;comparing-dev-random-speed-linux-bsd.h...</a> [1] <a href="https:&#x2F;&#x2F;superuser.com&#x2F;questions&#x2F;359599&#x2F;why-is-my-dev-random-so-slow-when-using-dd" rel="nofollow">https:&#x2F;&#x2F;superuser.com&#x2F;questions&#x2F;359599&#x2F;why-is-my-dev-random-...</a> [2] <a href="https:&#x2F;&#x2F;developer.apple.com&#x2F;videos&#x2F;play&#x2F;wwdc2020&#x2F;10686&#x2F;" rel="nofollow">https:&#x2F;&#x2F;developer.apple.com&#x2F;videos&#x2F;play&#x2F;wwdc2020&#x2F;10686&#x2F;</a>

It didn’t take months, the time I did it (running Docker on a Pinebook, which was not a great experience). It took a couple of hours to flip some base images away from Alpine, as Debian already has a load of ARM packages built.

So many wrong assumptions ...<p>1. If emulating aarch64 (arm64) on x86_64 is 6x slower (on your system, btw, it&#x27;s not an universal constant), it doesn&#x27;t mean emulating x86_64 on aarch64 will be 6x slower. It&#x27;d probably be worse, or at least that&#x27;s my gut feeling.<p>2. Generic container images like the Ubuntu mentioned usually have aarch64 (arm64) support, so running the x86_64 image makes no sense for the presented use-case.<p>3. You won&#x27;t be able to use most software because they don&#x27;t release ARM binaries ... and the example uses `wget` &amp;&amp; `tar xf`, with no binary signature check. As someone who has been porting stuff from x86_64 to aarch64 for a couple of years, I admit I&#x27;ve seen this pattern frequently. The most obvious solution is to build from sources, which would have been better off on x86_64 too, instead of fetching a prebuilt (and unverified) binary from the internet. Maybe there are some CPU flags the compiler could notice and apply optimizations which are not included in the prebuilt binary.<p>I&#x27;m not an Apple fan and I&#x27;m certainly not a fan of cross-architecture development either. I do agree with the general idea behind the article, however I find it a bit hand wavy.

Are there any excited embedded developers in the crowd? I have done a little embedded work and cross compiling has always been a huge pain in the ass to setup. I know some people have even gone as far as purchasing expensive niche workstations with ARM CPUs specifically to avoid this problem. I feel like having a mainstream ARM platform like the MBP will make compiling software for ARM-based single board computers a breeze.

My 2017 Macbook pro still has quite a lot of life in it, but it seems unlikely I will replace it with another Mac in a few years. Before the Mac I had a Lenovo X1 Carbon running Linux and it was great; and even then was a better development environment in some ways (docker has better filesystem performance, pacman is much better than homebrew). I do use some audio processing applications and my kids play a few games that do not run at all on Linux. I may try WSL again instead of going straight back to Linux, but its hard to imagine Mac will be the best OS for me.

If the worst of every possible thing happens <i>and</i> you avoid the most obvious solutions <i>and</i> one is very, very slow, then yes, you&#x27;re right to worry.<p>Or, you could use already-extant Debian ARM releases and spend minutes rather than months switching over.

I&#x27;d like to advocate for remote development environments. Most of my day is spent typing into a tmux session on a cloud-hosted box. (I picked up a Magic Keyboard for my 11&quot; iPad Pro, and thanks to Blink it&#x27;s a great glass terminal. It&#x27;s not going to work if you&#x27;re debugging let&#x27;s say a React app, but I&#x27;ve been very happy on it the last several days churning out Golang.)<p>Running stuff on your laptop makes it run slow, get hot, and burn battery. I&#x27;ve considered getting a small x86 or ARM media appliance as a (physically local) remote server for when I can&#x27;t count on an Internet connection. A media PC costs how much? The big holdup has been the tyranny of choice I&#x27;m confronted with. (Suggestions are welcome!)<p>I think very few people would be surprised if the coming of ARM Macs will, along with AWS&#x27;s ARM moves (and Microsoft&#x27;s), drive acceptance and adoption of ARM-based server computing. The mechanism won&#x27;t be anything formal, just the vague pressure that comes from people wanting their programs and libraries to compile locally.

<p><pre><code> I would expect about a 5x slowdown running Docker images. Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same architecture on the host as the guest, and are about about 1x - 2x as slow as running natively. Since you&#x27;re running ARM Mac, these hypervisors can only run ARM Linux. They can&#x27;t run x86_64 Linux. What will happen instead? These tools will fall back on emulators. </code></pre> Most of the software I run in Docker already supports ARM. I&#x27;d imagine that <i>a lot</i> of (most of?) us that use Docker do, too.

The loss of Boot Camp is huge. One of the reasons I develop on a Mac is so that I can use a single machine for all development (including macOS, iOS, and Windows development). Most of the time, developing for Windows on Parallels works fine, but there are some cases where it&#x27;s necessary to boot directly into Windows to test or debug adequately. I hope Apple is able to reach an agreement with Microsoft or at least continues shipping Intel-based Macs until then.

Apple will singlehandedly make 2021 &quot;the year of Linux on the desktop&quot;.

This is ridiculous. Your code will just have to support multiple architectures, which is very easy with modern languages and tooling.

I remember the days when having to switch between x86, ppc, sparc, etc was a thing (not to mention the many flavors of UN*X) and we survived. In fact I think it was more fun back before the x86&#x2F;Linux server domination. Architecture diversity is good.

I&#x27;m more convinced dropping dual boot and supporting virtualization is the right move.<p>Only the host OS is going to have the right drivers for the trackpad, wi-fi, GPU, power management, etc. etc. Through virtualization, the guest OS doesn&#x27;t have to worry about constantly evolving hardware models.<p>Virtualized OS performance is already very good, and USB passthrough has existed for a while. Snapshots are a godsend.<p>What won&#x27;t work are things like CUDA for eGPUs over Thunderbolt 3, and you&#x27;ll have to share disk and RAM with the host OS.<p>But for most use cases it&#x27;s probably the right choice. (This doesn&#x27;t address the author&#x27;s concern about moving away from x86.)

&gt; Why can&#x27;t you update the Docker image to also support ARM? You theoretically could switch your backend to run ARM Linux. However, this would take months - renting out ARM instances, re-building all repositories, and a tense switch over.<p>I don&#x27;t see why this would be so hard. If anything, I expect to see a massive upswing in things like AWS Graviton2 uptake, and a lot of common Docker images being built with ARM versions out of the box. It might be about a year or so, but eventually we&#x27;ll be able to just go ARM-native the whole way.<p>What Apple needs to do is make a first-class, WSL-tier implementation of Docker for Mac for ARM.

For my most recent project[1], I wanted to see if Amazon’s Graviton instances would be a good choice for my docker deployments (I was deploying MongoDB, an Express server, and several instances of the Janus WebRTC server). I was developing in Pop OS on an x86_64 desktop (since we’re gonna have to start specifying now) and found the toolchain around building ARM64 images to be pretty simple once I got it set up.<p>I benchmarked some `t2a.nano`s against some `a1.medium`s and found that the `nano`s were sufficient for my needs, so I went with them (they are cheaper than `a1.medium`s, even if the `a1`s have a better price-to-performance ratio).<p>I didn’t find it too difficult to rebuild any of these projects for cross-architecture usage. Even Janus, which has a TON of C&#x2F;C++ dependencies (some of which have to be compiled from a particular version of the source) easily built for ARM with no change in the Dockerfile.<p>So I kind of feel like OP is exaggerating the effort required to migrate servers to ARM. Sure it might be a hassle when you have tons of microservices, but you can move them incrementally, and most things recompile with no changes. And regardless of what architecture your dev machine is, you’ll want to be able to compile for and work with both architectures if you want to get the most out of the infrastructure on offer in 2020.<p>[1] Shameless plug: <a href="https:&#x2F;&#x2F;chrisuehlinger.com&#x2F;blog&#x2F;2020&#x2F;06&#x2F;16&#x2F;unshattering-the-audience-building-theatre-on-the-web-in-2020&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chrisuehlinger.com&#x2F;blog&#x2F;2020&#x2F;06&#x2F;16&#x2F;unshattering-the-...</a>

As a Linux tech, I welcome this Apple move honestly.<p>Having a proper competitor for x86&#x2F;x64 is a good thing.<p>The fact docker is slower on ARM (at the moment!) is mostly due to the lack of interests for optimizations.<p>With Apple starting to produce MacARM machines, and maybe more ARM servers in the wild, docker (and other platforms&#x2F;frameworks) will start to get more performant on ARM as well.

Do we know that Boot Camp isn’t supported by Big Sur? Or is it just that one can’t run an x86 OS on an ARM architecture? In other words - will I still be able to dual-boot into something like ARM-flavored Linux?

If you stay back on x86 virtualization will be slow, but if you jump to ARM this is great!<p>&gt; However, this would take months<p>(infomercial arms)<p>I&#x27;m sure it won&#x27;t make sense for everyone, but I&#x27;m just as sure it will make sense for many.

On the flip side, I guess ARM Macs will now allow the use of hypervisors for Android simulators, instead of a full hardware virtualization.<p>... thus making Android development better on Macs?

&gt; ec2 only offers 6 general-purpose ARM instance sizes<p>m6g, c6g, r6g each support 6 sizes for a total of 24

If you hadn&#x27;t sold yourself out of the free market, you would be able to <i>choose</i> what architecture machine you bought.

&gt; Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same architecture on the host as the guest, and are about about 1x - 2x as slow as running natively.<p>That doesnt sound right to me. Perhaps on IO bound tasks, if you are using emulated devices. On CPU bound tasks you should see near native performance.

There&#x27;s another unintended consequence of this virtualization - docker is already has very high CPU usage on my macbook, anywhere from 50-100%. Because of which it is always hot and toasty. This is caused already caused my screen to start deteriorating (<a href="https:&#x2F;&#x2F;www.ifixit.com&#x2F;Answers&#x2F;View&#x2F;567125&#x2F;Horizontal+line+on+bottom+of+MacBook+Pro+2017+(Due+to+overheating)" rel="nofollow">https:&#x2F;&#x2F;www.ifixit.com&#x2F;Answers&#x2F;View&#x2F;567125&#x2F;Horizontal+line+o...</a>) and the battery has degraded considerably too, even when I&#x27;m not coding on it and docker is shut down. This means a significant hit to the longevity of such devices as they not meant to be pushed so hard 40 hours a week. With ARM macs I see it getting even worse.

On the bright side, it looks like low-latency streaming is good enough that as long as you have internet connection, Boot Camp is not really necessary. This works well for me: <a href="https:&#x2F;&#x2F;shadow.tech&#x2F;usen&#x2F;" rel="nofollow">https:&#x2F;&#x2F;shadow.tech&#x2F;usen&#x2F;</a>

A lot of developer-centric focus discussion on how Docker would work (hint: it does), but VirtualBox is still pretty common in the sysadmin world and other industry circles. Moreover, there seems to be no way it will ever work. It will be interesting to see how that turns out.

Any company with a load of binaries built without any effort towards supporting cross platform builds that uses Docker is gonna have a bad day with this. They buy a bunch of new MacBooks and then find they can&#x27;t use them until they spend a few weeks porting everything.

My guess is that Apple will end up copying Microsoft and providing a WSL style Linux kernel &quot;shim&quot; into Darwin (pretty easy as it&#x27;s already UNIX) and use Rosetta2 to translate any x86_64 containers to aarch64). No need for a hypervisor.

I wonder if Apple will still allow a dual boot with a native (arm64 in this case) Linux

Once again -- Apple will do what the entire industry without Apple couldn&#x27;t do. In this case, force a migration to ARM-based servers, so that prod is running on the same architecture as the developer&#x27;s machine.<p>Apple is finally killing x86.

But linux can be run on arm natively. Moreover, most packages are also compiled for arm. So apt-get install will work just the same. I&#x27;m sure they will be able to target Apple&#x27;s specific arm chips when they come out.

If I were to describe my job, or programming in general, it might be &quot;problems that have no obvious solution&quot;. This is a sad article that just seems to be the opposite in spirit to &quot;Hacker&quot; ethos.

&gt; Should I get an ARM Mac? ...if you use virtualization often, I wouldn&#x27;t recommend it.<p>I use virtualization continuously, but not for anything that needs to be as fast as possible.<p>I won’t hesitate to get an ARM Mac once I can run x64 Windows VMs on it. (Presumably VMWare Fusion or Parallels, and for once I won’t feel ripped off by the upgrade pricing.)<p>Docker on Mac doesn’t work that well today, so I don’t have any workflows that depend on it.

This is silly. Most stacks will have counterparts on both architectures. Just run CI with the same configurations as prod. Problems due to differences will be rare for most stacks -- modern languages are well defined and run against extensive spec tests on all major platforms -- and will smooth out over time.

All my Docker stuff is multi-arch these days. Here’s one of my sample pipelines, written precisely to show how easy it is: <a href="https:&#x2F;&#x2F;github.com&#x2F;rcarmo&#x2F;azure-pipelines-multiarch-docker" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rcarmo&#x2F;azure-pipelines-multiarch-docker</a>

I&#x27;m not a heavy docker user but why can&#x27;t I develop docker containers on Arm (as native containers, no emulation) and deploy to x86_64? Or vice versa? I understand that some packages are binary only and wouldn&#x27;t be necessarily available for Arm, especially initially, but the majority should be.

It will be extremely interesting to see what the dev&#x27;s say with the test boxes. Curious if there&#x27;s any magical fixes like switching over to ARM Linux, since the software of an image would likely be compiled for x86 I really doubt it...<p>Perhaps this will spur some people over to running ARM servers in the cloud...

It’s still early days. bhyve which is likely what they’re using or whatever the hypervisor is will just run arm docker images — unless you have a hard x86 dependency many of our http micro services should run just fine on arm images at least my workflow will be just fine.

&gt; Why can&#x27;t you update the Docker image to also support ARM? You theoretically could switch your backend to run ARM Linux. However, this would take months<p>No need to take months. `docker buildx` can build multi-arch images without using real ARM instances.

I use docker every day and I guess I’m just not worried about this. The container pushes come from a CI host, so I’m not worried about compatibility.<p>And it’s 20% slower? Well, most of the build time is slow for all sorts of reasons. I honestly don’t think I’ll even notice.

this is why intel should be scared - very scared - that the PC world seems to be converting to ARM en masse. PCs might be a relatively small fraction of intel’s total sales, but it’s the second order effects they should be worried about. if it becomes less convenient for developers on ARM machines to develop and deploy software to x86 cloud architecture, they will begin to demand that the cloud architecture be shifted to ARM as well.

Didn&#x27;t think of that. If running Docker is 10 times slower on ARM and virtual box doesn&#x27;t support the architecture at all, this might indeed end developers using Mac.

We have qemu on arm. It’s fast enough to software render half-life. Servers will follow desktop, so in a few years a lot of things are arm

Hey guys... we have done emulation before, and it&#x27;s not nearly so bad.<p>Also, there ARM images for Docker. You don&#x27;t HAVE to run x86-64 binaries.

Develop on the platform you want to deploy to.

I guess someone will port Docker to JVM, and use the JVM optimized to whatever ARM processor there will be.

Apple has their own virtualization offering to share and they are being quite coy about

Has Apple mentioned something about Thunderbolt 3 or USB 4 on this new Macs?

I had an ARM based desktop in 1988. Be thrilled to have one again.

Hypervisors are by no means 1-2x slower. Testing I&#x2F;O is not testing the performance of a hypervisor. It&#x27;s testing the I&#x2F;O stack.

Tl;dr they want to run x86 linux for their own reasons rather than arm.

Mac book was never really a dev platform. Maybe for front or nodejs, or definitly for native apple apps, but seriously, brew and so are so subpar.

When I ask Mac-loving developers, why they chose to run MacOS when developing non-MacOS software, they used to give me good reasons.<p>I think their reasoning is no longer valid. The hardware has gotten worse (keyboard, touchbar), the OS has gotten more hostile, meanwhile the state of Linux on Laptop has gotten a lot better. So... I used to understand them, but I no longer do.<p><a href="https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;photos&#x2F;images&#x2F;newsfeed&#x2F;001&#x2F;016&#x2F;674&#x2F;802.jpg" rel="nofollow">https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;photos&#x2F;images&#x2F;newsfeed&#x2F;001&#x2F;016&#x2F;674&#x2F;802...</a>