TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Serverless Map/Reduce

259 点作者 emilong超过 8 年前

17 条评论

ralusek超过 8 年前
This is a screenshot of my google search from 2 days ago:<p><a href="http:&#x2F;&#x2F;i.imgur.com&#x2F;BNAcSsn.png" rel="nofollow">http:&#x2F;&#x2F;i.imgur.com&#x2F;BNAcSsn.png</a><p>I&#x27;ve been using Lambda quite a bit, I think it&#x27;s SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invoking a lambda directly from your code.<p>Imagine a case where you had to parse a million documents with a relatively expensive computation, let&#x27;s say 250MS per document. Maybe you have a solid machine with a few cores that&#x27;s running your server, but even then you can&#x27;t have the server cpu locked for so long, so naturally you&#x27;d need some sort of worker server set up. With a good machine and multiple cores, maybe you get 8 running at once. With a lambda, you can forego the worker server altogether. Just invoke a million lambdas directly from your application server, completely parallelized.<p>Theoretically, you&#x27;ve taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.
评论 #12874436 未加载
评论 #12876667 未加载
评论 #12878282 未加载
评论 #12878869 未加载
评论 #12874450 未加载
评论 #12876681 未加载
stochastician超过 8 年前
Author of the figures used in the blog post here. We wrote <a href="https:&#x2F;&#x2F;github.com&#x2F;ericmjonas&#x2F;pywren" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ericmjonas&#x2F;pywren</a> somewhat on a lark, because it seemed to fit well with our research goals and it&#x27;s fun to push systems to their limit. I&#x27;m now a total serverless convert! I&#x27;d love more collaborators and feedback, the goal is to make these sorts of computations as easy as possible for python developers, especially on the scientific computing side of things.
danso超过 8 年前
OT: I teach computational methods and even as much as I dislike teaching&#x2F;conflating it with web dev, I have included &quot;let&#x27;s build a web app&quot; because students like building and deploying a thing, and because Heroku has a free tier.<p>I&#x27;ve considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with controlled access via IAM...but is there a way to set up rate-limiting, ideally through a policy? That is, shut an IAM down if a student accidentally invokes a million processes? Or, for that matter, limiting the storage capacity of a S3 bucket?
评论 #12877253 未加载
评论 #12877326 未加载
评论 #12880993 未加载
评论 #12877456 未加载
评论 #12877212 未加载
stcredzero超过 8 年前
I wonder if something like AWS Lambda could be applied to multiplayer games? It seems like game-loop based games would be a good domain for such a programming model. The entire game could be expressed as a function that turns tick N into tick N+1. Such a function would be composed of many other functions, of course. So for example, there would also be a function that took as an argument the player at time N and gave the player at time N+1.<p>Such a model would allow infrastructure developers to abstract away most of the concerns around networking, collisions, security, etc., and let game developers concentrate their efforts on simply making the game.<p>I currently have a game server cluster written in Golang, where the locations are instantiated with an idempotent request operation. It doesn&#x27;t matter if a particular location-instance exists at a particular moment. It&#x27;s sufficient for the &quot;master control&quot; server to only approximately know the loads of the different cluster server processes. My experience leads me to believe that something like AWS Lambda, but optimized for implementing game loops would work well, so long as game developers could get their heads around pure functional programming and implement with soft real-time requirements in mind. (John Carmack already advocates the use of pure functions, and game devs in general already do the latter.)<p><a href="http:&#x2F;&#x2F;www.emergencevector.com" rel="nofollow">http:&#x2F;&#x2F;www.emergencevector.com</a>
评论 #12874666 未加载
评论 #12875821 未加载
lucd超过 8 年前
How does it compare to 3 years old Joyent&#x27;s Manta ? AFAIK it was especially designed for this kind of purposes. The processing is made directly on the servers storing the data..
评论 #12874780 未加载
thinkloop超过 8 年前
The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage.<p>But what&#x27;s the solution for structured data? DynamoDB is the obvious main candidate, but it&#x27;s billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict.<p>Is there a good solution for running massively parallel lamdas on stuctured data?
评论 #12877469 未加载
评论 #12877471 未加载
partycoder超过 8 年前
I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved.<p>It&#x27;s like saying deathless meat, because someone else killed the animal you are consuming.
评论 #12876632 未加载
评论 #12877147 未加载
评论 #12875661 未加载
评论 #12877450 未加载
评论 #12876527 未加载
评论 #12875631 未加载
plandis超过 8 年前
I&#x27;ve always had one big question about Lambda. Is it really worth the cost you get for the convienience of it?<p>Is anyone using it in production that can comment?
评论 #12878315 未加载
评论 #12878290 未加载
eistrati超过 8 年前
Saw the presentation last week at ServerlessConf in London and it really looks very promising. The cost behind this solution is what will really make me check this out :)<p>P.S. Quoting the author: &quot;As you can see for these queries, the reference implementation performs reasonably well; it&#x27;s nowhere near Redshift performance for the same queries, but for the price it really can&#x27;t be beat today&quot;
dnackoul超过 8 年前
Does anyone have experience building mobile backends in Lambda? I was looking at an API Gateway &#x2F; Lambda &#x2F; Amazon RDS stack for building a central data store and was wondering what people&#x27;s experience with that setup is?
评论 #12876279 未加载
c-smile超过 8 年前
About the site: quite hard to read - almost white text on white background.
评论 #12875441 未加载
boulos超过 8 年前
Note: the underlying comparison to other systems is from a 2014 blogpost [1] which suggest they used the m2.4xlarge series of EC2 VMs (which were Nehalem class parts from 2010). Nehalem vs Haswell or Broadwell (the likely parts underlying Lambda) is a pretty big jump.<p>Disclosure: I work on Google Cloud, but I&#x27;m just pointing out a fact ;).<p>[1] <a href="https:&#x2F;&#x2F;amplab.cs.berkeley.edu&#x2F;benchmark&#x2F;" rel="nofollow">https:&#x2F;&#x2F;amplab.cs.berkeley.edu&#x2F;benchmark&#x2F;</a>
mallya16超过 8 年前
Implementation guide for Serverless MapReduce: <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;compute&#x2F;ad-hoc-big-data-processing-made-simple-with-serverless-mapreduce&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;compute&#x2F;ad-hoc-big-data-process...</a>
willcodeforfoo超过 8 年前
I wonder if Amazon will ever open Lambda up to any Docker image? (I know it&#x27;s possible to run binaries, but its a bit of a pain to compile with the Amazon AMI, etc.) Being able to have a bunch of `docker run` with any image would be pretty powerful.
评论 #12876472 未加载
评论 #12879017 未加载
frenchhacker超过 8 年前
I guess the example assumes the data is already somehow in AWS. How is the total cost affected if I wanted to run this setup on a 10TB dataset?
elcct超过 8 年前
Is there any AWS Lambda equivalent that could be deployed on bare metal?
评论 #12876719 未加载
评论 #12877225 未加载
评论 #12876281 未加载
评论 #12876282 未加载
评论 #12876283 未加载
amelius超过 8 年前
If it doesn&#x27;t run on a server, then what does this plumbing-work run on? Clickbait name?
评论 #12874210 未加载
评论 #12874242 未加载
评论 #12874235 未加载
评论 #12874374 未加载
评论 #12874193 未加载