TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Apache Arrow Flight: A Framework for Fast Data Transport

128 点作者 stablemap超过 5 年前

7 条评论

fulafel超过 5 年前
It&#x27;s interesting how much faster your laptop SSD is compared to these high end performance oriented systems. Keeping in mind that the localhost&#x2F;tls-disabled number is a high bound. (Not singling out Arrow by any means, most others are slower. )<p>I wonder which came first, the petering off of wired network hardware perf improvement, or the software bottlenecks that become obvious if we try to use today&#x27;s faster networks. 100 Mb ethernet came in 1995, gigE in 1999, 10 gigE in 2002 and gained adoption in a few years.. on that track we should have had 100gigE in 2006 and seen it in servers in 2008 &#x2F; workstations in 2010. And switches &#x2F; routers should have seen terabit ethernet in 2010. Today&#x27;s servers(X) seem to be at about 25 GBe, and with multicore that&#x27;s just 1-2 gigabits per core.<p>(X) according to <a href="https:&#x2F;&#x2F;www.supermicro.com&#x2F;products&#x2F;system&#x2F;1U&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.supermicro.com&#x2F;products&#x2F;system&#x2F;1U&#x2F;</a>
评论 #21269016 未加载
jumpingmice超过 5 年前
More people should try high performance services with non-traditional protobuf implementations. The fact that every language has a generated parser in no way preclude you from parsing them yourself. Hand-rolled serialization of your outbound messages can also be really fast, and the C++ gRPC stack will just accept preformatted messages and put them on the wire. Finally the existence of gRPC itself should not make you feel constrained against implementing the entire protocol yourself. It’s just HTTP&#x2F;2 with conventional headers.
评论 #21262354 未加载
wodenokoto超过 5 年前
A bit off topic, but since this is implemented using gRPC, I’d like to ask, what is RPC and how does one make an (g)RPC call?<p>My understanding is it’s a binary alternative to JSON&#x2F;REST API and all google cloud platform services uses it, however, since I have not managed to figure out how to do a single interaction with RPC against gcp (or any other service), I am wondering if my understanding is completely wrong here.
评论 #21267668 未加载
riboflavin超过 5 年前
Also, for more on Arrow and Flight: <a href="https:&#x2F;&#x2F;www.dremio.com&#x2F;understanding-apache-arrow-flight&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.dremio.com&#x2F;understanding-apache-arrow-flight&#x2F;</a>
algorithmsRcool超过 5 年前
Are there any thoughts about where compression fits into this model?<p>I know networks are getting very fast but with this size of data I wonder if there are realizable gains left with modern algorithms like Snappy.
评论 #21266400 未加载
RocketSyntax超过 5 年前
We are struggling with reliability when using mounting solutions for big data in S3. Would this help?
评论 #21263584 未加载
评论 #21262218 未加载
评论 #21263619 未加载
maximente超过 5 年前
meta: the https version works fine:<p><a href="https:&#x2F;&#x2F;arrow.apache.org&#x2F;blog&#x2F;2019&#x2F;10&#x2F;13&#x2F;introducing-arrow-flight&#x2F;" rel="nofollow">https:&#x2F;&#x2F;arrow.apache.org&#x2F;blog&#x2F;2019&#x2F;10&#x2F;13&#x2F;introducing-arrow-f...</a>