Teleforking a Process onto a Different Computer

Congratulations! You have just reinvented the core idea of UCLA&#x27;s LOCUS distributed computing project from 1979. <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LOCUS" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LOCUS</a><p>Reinventing LOCUS also has a strong heritage. Bell Lab&#x27;s Plan 9, for example, did so in part in the late 1980s.<p>While never a breakout commercial success, tele-forking and its slightly more advanced cousins machine-to-machine process migration and cluster-wide process pools intrigued some of the best minds in distributed computing for 20+ years.<p>Unfortunately &quot;it&#x27;s complicated&quot; to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) that are important to scale up the idea.

In the late 90s I attended a tour at Holland Signaal, a dutch defense company producing radar and anti-missile systems.<p>I remember vividly how they demonstrated an unbreakable process. They had a computer running a process and no matter what happened to that computer, the next one would flawlessly continue the process down to the cycle, with no change or corruption or skipping a beat.<p>It may very well be that this is actually not very difficult, but it seemed difficult and impressive.<p>Perhaps more shocking were ultra high resolution radar screens, some 3 generations ahead of anything I had seen in the consumer space, showing an incredible visualization of the air space, live. Showing exactly which plane is where, the model&#x2F;type, age, fuel on board, hostile&#x2F;friendly, all of it.<p>They even had a &quot;situation room&quot; with a holodeck chair in the middle, full of controls. The entire room was covered in wall-size screens basically showing the air space of the entire country, being live analyzed.<p>Sounds very 2022, not 1998.

Both MOSIX and openMOSIX supported fork()ing to another node on the network. <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;MOSIX" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;MOSIX</a>

Doesn&#x27;t Erlang support these ideas of distributed computing .... And if I recall correctly Clipper supported remote execution of objects, or sharing object code in a distributed fashion.

Always worth a reread: <a href="https:&#x2F;&#x2F;joearms.github.io&#x2F;published&#x2F;2013-11-21-My-favorite-erlang-program.html" rel="nofollow">https:&#x2F;&#x2F;joearms.github.io&#x2F;published&#x2F;2013-11-21-My-favorite-e...</a>

For what it&#x27;s worth, at least for HPC-ish distributed computing, this sort of thing turns out not to be terribly worthwhile. We have a standard for distribution of computation, shared memory, i&#x2F;o, and process starting in MPI (and, for instance, DMTCP to migrate the distributed application if necessary, though I think DMTCP needs a release).<p>I don&#x27;t know what its current status is, but the HPC-ish Bproc system has&#x2F;had an rfork [1]. Probably the most HPC-oriented SSI system, Kerrighed died, as did the Plan 9-ish xcpu, though that was a bit different.<p>1. <a href="https:&#x2F;&#x2F;www.penguinsolutions.com&#x2F;computing&#x2F;documentation&#x2F;scyld-clusterware&#x2F;7&#x2F;admin-guide&#x2F;ref-guide&#x2F;libraries&#x2F;bproc&#x2F;bproc-rfork.html" rel="nofollow">https:&#x2F;&#x2F;www.penguinsolutions.com&#x2F;computing&#x2F;documentation&#x2F;scy...</a>

Off-topic, but google-fu is failing me: wasn&#x27;t there a similar project posted here a while back, that instead shipped a binary + a virtual file system that mirrored all local resources to a remote host so that the entire process could run there? IIRC the author demonstrated it running an ffmepg transcode?<p>edit: Got it, after a while: <a href="https:&#x2F;&#x2F;github.com&#x2F;Overv&#x2F;outrun" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Overv&#x2F;outrun</a>

(2020) and discussed at the time here: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22987747" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22987747</a>

I&#x27;ve been playing with adjacent ideas in WebAssembly. The idea is that you can write a function that sends and receives messages of arbitrary Rust types, compile it into WebAssembly, and then snapshot and restore it (including across space and time) as you like.<p>It&#x27;s very experimental, but I got as far as being able to freeze a JavaScript interpreter (compiled to WebAssembly) and restore it later. The code is here: <a href="https:&#x2F;&#x2F;github.com&#x2F;drifting-in-space&#x2F;wasmbox" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;drifting-in-space&#x2F;wasmbox</a>

The Cloud Haskell project and language is likely the only one to get this right, thanks to strictly enforced purity. It&#x27;s much simpler to understand, absence global mutable state, whether it&#x27;s safe and possible to serialize a closure and run it somewhere else. (fork(2) being a closure by another name.)<p>In almost all other languages there&#x27;s just no way to know if a closure is holding on to a file descriptor.<p>Critics may say the Haskell closures could contain `unsafePerformIO`, but as the saying goes: now you have two problems.

I can&#x27;t state with absolute certainty but I seem to recall OpenVMS clusters back in the 90&#x27;s transparently supporting &quot;portable&quot; processes across cluster nodes.

That used to be in some UNIX variants, such as UCLA Locus and the IBM derivatives of that. But it never got to be a Linux thing.

If I am understanding the article correctly, BEAM does this for Elixir and Erlang (and other BEAM languages) out of the box.

Termite Scheme did something like this for Scheme programs. It combines Erlang-style concurrency with process migration.<p><a href="http:&#x2F;&#x2F;www.iro.umontreal.ca&#x2F;~feeley&#x2F;papers&#x2F;GermainFeeleyMonnierELSW05.pdf" rel="nofollow">http:&#x2F;&#x2F;www.iro.umontreal.ca&#x2F;~feeley&#x2F;papers&#x2F;GermainFeeleyMonn...</a>

You could do what CRIU does, but leave the original process running, and arrange to have a composite {instance ID, PID} as the return value (different on both sides).<p>(Trivia: in Unix VI, fork() returns both PIDs, and the C library stub arranges to return 0 to the child and the child&#x27;s PID to the parent.)

This is fun. Reminded me of mincemeat (<a href="https:&#x2F;&#x2F;github.com&#x2F;ziyuang&#x2F;mincemeatpy" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ziyuang&#x2F;mincemeatpy</a>), which actually serializes the mapping functions and sends them to workers.

I don&#x27;t think anyone has mentioned Sprite <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Sprite_(operating_system)" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Sprite_(operating_system)</a><p>Sprite called this &quot;process migration&quot;.

Lambda&#x2F;Cloud Functions are starting to converge on this idea. It will eventually get streamlined and ergonomic enough that it appears you&#x27;re executing an expensive or async function locally, except you aren&#x27;t.

this is a fun hack. it would be interesting to look at some real world workloads and compare whether this sort of init once ship initialized memory image everywhere style is faster than just initializing everywhere.

This is really cool, Distributed Unix could do this IIRC.

I also had this idea but I called it networkfork.

isn’t this exactly what the vm migration in cloud is?

This sounds like Kafka, but more low level.