TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Ray: A Distributed Framework for Emerging AI Applications

134 pointsby mlerneralmost 4 years ago

12 comments

sillysaurusxalmost 4 years ago
I used Ray to train a massive GPT model by putting each layer on a separate TPU. Ray was able to send all the gradients back and forth as needed.<p>It scaled fine up to 33 TPUs (i.e. 33 layers).<p>Ray is impressive as hell.<p>By the way, I didn&#x27;t write the code to do any of that. kindiana, aka &quot;the guy that wrote GPT-J&quot;, also happened to write this:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;kingoflolz&#x2F;swarm-jax" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kingoflolz&#x2F;swarm-jax</a><p>I just ran it and it worked. Which is extraordinarily unusual for TPUs, historically speaking.<p>I&#x27;m pushing my luck at this point, since it&#x27;s crossing the line from enthusiasm to spam. But if you want to try out Ray on TPUs like I did here, I posted a (massive) amount of detail on how to get started, and why: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27728225" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27728225</a><p>That&#x27;s the last I&#x27;ll be mentioning it for some time, though.<p>Ray + JAX is such a killer combo.
评论 #27733575 未加载
评论 #27734979 未加载
Rich_Morinalmost 4 years ago
Although it&#x27;s early days, José Valim and some other folks are working on adding AI-related capabilities to Elixir and the (Erlang) BEAM. See &quot;Introducing Nx&quot; (<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fPKMmJpAGWc" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fPKMmJpAGWc</a>) for an intro.<p>Given that they already have a robust Actor model as a base to work from, it occurs to me that they may be able to use some of Ray&#x27;s ideas as they go along...
Tenokealmost 4 years ago
I&#x27;ve used Ray for about a year (typically for thousands of ML tasks, spread across ~48-120 cores simultaneously) and it&#x27;s a pleasure to use at least using the basic API. Admittedly, I had problems when trying to use some of the more advanced approaches but I didn&#x27;t really need them and I can definitely recommend it since the performance is great.
评论 #27731855 未加载
reubenbondalmost 4 years ago
Given how Ray &quot;provides [...] exactly-once semantics&quot; for its actors, you could draw similarities between it and workflow-as-code frameworks such as <a href="https:&#x2F;&#x2F;temporal.io" rel="nofollow">https:&#x2F;&#x2F;temporal.io</a>. The way that Ray splits up actors and tasks looks similar to Temporal&#x27;s Workflows + Activities split: Workflows (Ray actors) contain orchestration logic and have their method calls&#x2F;results durably logged. Activities (Ray tasks) perform the expensive computations and any interaction with external systems and are not durably logged.<p>If you&#x27;re in the .NET ecosystem or interested in distributed systems in general, you may like Orleans (<a href="https:&#x2F;&#x2F;github.com&#x2F;dotnet&#x2F;orleans" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dotnet&#x2F;orleans</a>), which I work on at Microsoft. Orleans contributes the Virtual Actor model which other modern actor frameworks are starting to adopt since it is well suited for the hectic, failure-prone environment of distributed systems (which those so-called Cloud Native Apps live in). The Ray paper linked from the article (<a href="https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;osdi18-moritz.pdf" rel="nofollow">https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;osdi18-moritz.pdf</a>) discusses some similarities. Slight correction on the paper: it states that &quot;For message delivery, Orleans provides at-least-once [...] semantics&quot;. It&#x27;s at-most-once. At-least-once messaging semantics (usually implemented via automatic retries) aren&#x27;t ideal for these kinds of systems, in my opinion.
评论 #27732204 未加载
ramozalmost 4 years ago
I spent the past year and a half deploying a distributed backend for bert-like models &amp; we ultimately chose a K8s architecture &amp; &quot;precise&quot; affinity mapped out, which is still hard due to cpu pinning issues. On the frontend-api, Golang gives us the ability to distribute &amp; split requests coming in (10-20M &#x2F; day &amp; batch size averaging ~3K which splits into 50 due to model constraints). Embeddings are stored on those nodes, local ssds. Those nodes are only a handful. Models run on 2 pools, 1 dedicated and one preemptible (most nodes here) which gives us cost optimization and scheduling is simplified due to K8s. We have anywhere from 120-300 of these high compute nodes.<p>Wondering if anyone has similar deployments and migrated to Ray. We&#x27;ve evaluated it but can&#x27;t afford a large migration at this point &amp; would also need to test quite a bit &amp; rebuild our whole automation for infra and apps.<p>Really interested though as the infrastructure isn&#x27;t cheap and every time the model updates we are basically re-architecting it. Right now we are moving everything away from python (gunicorn&#x2F;flask, and MKL) to Golang as we can get better efficiencies with data serialization (numpy ops are the biggest time eaters right now ... model input vectors constructed from flatbuffers)
评论 #27732293 未加载
评论 #27731713 未加载
wolfium3almost 4 years ago
There was a recent talk at PyCon US 2021 on this :)<p>TALK &#x2F; SangBin Cho &#x2F; Data Processing on Ray [<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=DNLqvdov_J4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=DNLqvdov_J4</a>]
flakinessalmost 4 years ago
At a glance, Ray is a re-invention (or rebranding) of distributed object system plus agent, which was popular around &#x27;90-&#x27;00. Things like Java RMI and CORBA (remember?) was part of the trend, until REST killed them all.<p>On top of the distributed object foundation, Ray added a ML-oriented twists like efficient numerical data transfer with Apache Arrow, and shifted focus from (classic) agent system to RL, or general distributed ML training in general, accompanied by the Python-first approach - which simplifies a lot of things compared to traditional, often language-agnostic distributed objects.<p>I&#x27;m not claiming Ray is not novel. Rather, my point is that what a dated idea needs to come back may be just some relevant-today twists like these. I think Ray is good demonstration of possibility of such old new things.
blueyesalmost 4 years ago
Our team has used Ray for more than two years across several versions. It makes a lot of things easy that were not, and is especially adapted for our purposes, which include training and deploying a lot of reinforcement learning policies. The AnyScale team is very responsive on the support Slack, fwiw.
phissenschaftalmost 4 years ago
Great work and kudos to the Ray team! It&#x27;s definitely a fresh look with a lot of lessons learned from previous generations (e.g. spark).<p>There are a few nice features I wish Ray would eventually get to.<p>On the user experience side, it would be nice to have task level logs: often time it&#x27;s easier for users to reason at task level, especially the task is a facade that triggers other complicated library&#x2F;subprocess calls.<p>For the scheduler, if there&#x27;s more native support for sharded&#x2F;bundled&#x2F;partitioned tasks and <a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;blog&#x2F;products&#x2F;gcp&#x2F;no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;blog&#x2F;products&#x2F;gcp&#x2F;no-shard-left-beh...</a>
robertnishiharaalmost 4 years ago
Hi all, I&#x27;m one of the authors of Ray, thanks for all the comments and discussion! To add to the discussion, I&#x27;ll mention a few conceptual things that have changed since we wrote the paper.<p>*Emphasis on the library ecosystem*<p>A lot of our focus is on building an ecosystem of libraries on top of Ray (much, but not all, of the focus is on machine learning libraries). Some of these libraries are built natively on top of Ray such as Ray Tune for scaling hyperparameter search (<a href="http:&#x2F;&#x2F;tune.io" rel="nofollow">http:&#x2F;&#x2F;tune.io</a>), RLlib for scaling reinforcement learning (<a href="http:&#x2F;&#x2F;rllib.io" rel="nofollow">http:&#x2F;&#x2F;rllib.io</a>), Ray Serve for scaling model serving (<a href="http:&#x2F;&#x2F;rayserve.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;rayserve.org&#x2F;</a>), and RaySGD for scaling training (<a href="https:&#x2F;&#x2F;docs.ray.io&#x2F;en&#x2F;master&#x2F;raysgd&#x2F;raysgd.html" rel="nofollow">https:&#x2F;&#x2F;docs.ray.io&#x2F;en&#x2F;master&#x2F;raysgd&#x2F;raysgd.html</a>).<p>Some of the libraries are popular libraries on their own, which now integrate with Ray such as Horovod (<a href="https:&#x2F;&#x2F;eng.uber.com&#x2F;horovod-ray&#x2F;" rel="nofollow">https:&#x2F;&#x2F;eng.uber.com&#x2F;horovod-ray&#x2F;</a>), XGBoost (<a href="https:&#x2F;&#x2F;xgboost.readthedocs.io&#x2F;en&#x2F;latest&#x2F;tutorials&#x2F;ray.html" rel="nofollow">https:&#x2F;&#x2F;xgboost.readthedocs.io&#x2F;en&#x2F;latest&#x2F;tutorials&#x2F;ray.html</a>), and Dask for dataframes (<a href="https:&#x2F;&#x2F;docs.ray.io&#x2F;en&#x2F;master&#x2F;dask-on-ray.html" rel="nofollow">https:&#x2F;&#x2F;docs.ray.io&#x2F;en&#x2F;master&#x2F;dask-on-ray.html</a>). While Dask itself has similarities to Ray (especially the task part of the Ray API), Dask also has libraries for scaling dataframes and arrays, which can be used as part of the Ray ecosystem (more details at <a href="https:&#x2F;&#x2F;www.anyscale.com&#x2F;blog&#x2F;analyzing-memory-management-and-performance-in-dask-on-ray" rel="nofollow">https:&#x2F;&#x2F;www.anyscale.com&#x2F;blog&#x2F;analyzing-memory-management-an...</a>).<p>Many Ray users start using Ray for one of the libraries (e.g., to scale training or hyperparameter search) as opposed to just for the core system.<p>*Emphasis on serverless*<p>Our goal with Ray is to make distributed computing as easy as possible. To do that, we think the serverless direction, which allows people to just focus on their code and not on infrastructure, is very important. Here, I don&#x27;t mean serverless purely in the sense of functions as a service, but something that would allow people to run a wide variety of applications (training, data processing, inference, etc) elastically in the cloud without configuring or thinking about infrastructure. There&#x27;s a lot of ongoing work here (e.g., to improve autoscaling up and down with heterogeneous resource types). More details on the topic <a href="https:&#x2F;&#x2F;www.anyscale.com&#x2F;blog&#x2F;the-ideal-foundation-for-a-general-purpose-serverless-platform" rel="nofollow">https:&#x2F;&#x2F;www.anyscale.com&#x2F;blog&#x2F;the-ideal-foundation-for-a-gen...</a>.<p>If you&#x27;re interested in this kind of stuff, consider joining us at Anyscale <a href="https:&#x2F;&#x2F;jobs.lever.co&#x2F;anyscale" rel="nofollow">https:&#x2F;&#x2F;jobs.lever.co&#x2F;anyscale</a>.
评论 #27733451 未加载
orfalmost 4 years ago
How does this compare to Dask?
评论 #27733436 未加载
评论 #27741969 未加载
maxthegeek1almost 4 years ago
Honestly looks really cool, I want to try using Ray.