TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Ask HN: Is there a Hadoop alternative built on non-Java technologies?

2 点作者 tmbsundar超过 9 年前
For distributed or "big data" scale computation Hadoop has been so successful that, one is not able to find information about any alternatives that exist. For one, it will be good if any of the ML family of languages/ LISP etc., had an viable alternative. There are Python bindings that exist and DSLs like Pig/Hive simplify things a bit. But beyond that, without knowledge of Java it's hard to navigae the Hadoop ecosystem. Hence this question.

1 comment

mindcrime超过 9 年前
It depends on how tightly you define &quot;Hadoop alternative&quot; and exactly what you&#x27;re trying to do. But generally speaking, the answer is &quot;yes&quot;. There are technologies for doing large-scale distributed computation on clusters of commodity computers that don&#x27;t require Java. In fact, many (most?) of them predate Java. You can, for example, build a Beowulf cluster[1] using something like Rocks[2] and program using MPI[3], OpenMP[4] or PVM[5].<p>The one big difference is that you don&#x27;t get the advantage of the (relatively speaking) easy to understand and code against Map-Reduce API. Programming in MPI requires a bit more low-level fiddling... something like the difference between, say, programming in C vs programming in Java. Which is, I suppose, quite appropriate since most MPI programming is done in C (or using the C bindings from another language).<p>There are also some Map-Reduce implementations natively written in C++ floating around out there.<p>Note that if you do something like a Beowulf cluster you still need a distributed file-system to serve the role HDFS plays in the Hadoop world.<p>There&#x27;s also this:<p><a href="https:&#x2F;&#x2F;gigaom.com&#x2F;2015&#x2F;02&#x2F;18&#x2F;google-open-sources-a-mapreduce-framework-for-c&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gigaom.com&#x2F;2015&#x2F;02&#x2F;18&#x2F;google-open-sources-a-mapreduc...</a><p>but I haven&#x27;t used it and don&#x27;t know much about it.<p>[1]:<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Beowulf_cluster" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Beowulf_cluster</a><p>[2]: <a href="http:&#x2F;&#x2F;www.rocksclusters.org&#x2F;wordpress&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.rocksclusters.org&#x2F;wordpress&#x2F;</a><p>[3]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Message_Passing_Interface" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Message_Passing_Interface</a><p>[4]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;OpenMP" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;OpenMP</a><p>[5]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Parallel_Virtual_Machine" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Parallel_Virtual_Machine</a>