TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Space Monkey dumps Python for Go

338 点作者 jtolds大约 11 年前

29 条评论

chimeracoder大约 11 年前
This article doesn&#x27;t address one of the biggest benefits I&#x27;ve noticed in switching from Python to Go.<p>I&#x27;ve been using Go for almost two years now. Like OP, I am&#x2F;was primarily a Python developer. Like OP, my first Go project was a time-sensitive rewrite[0] of a project from Python (tornado) to Go.<p>Even though I was an experienced Python developer, the Go rewrite was marginally (~20%) faster[1]. But the real benefit came from the <i>subsequent</i> development - refactoring, rearchitecting, and maintaining the project on an ongoing basis. Go was designed to make it easy to scale the maintenance[2] of software, and on this one axis, it absolutely blows every other language and environment I&#x27;ve used out of the water.<p>For a fresh project, I&#x27;d say Go is about 10% slower to write than the equivalent Python[3] for the average case. But the time&#x2F;cost savings are <i>very</i> quick to come thereafter.<p>[0] I would absolutely not recommend doing time-sensitive rewrites in general, but that decision was a separate matter.<p>[1] Some of this is due to the nature of rewrites in general, but the fact that it wasn&#x27;t slower to use a language I&#x27;d never used before says something about the language.<p>[2] Scaling software development as teams grow is <i>very</i> different from scaling software as users grow.<p>[3] Assuming comparable familiarity with both languages, which is rarely the case, of course
评论 #7629126 未加载
评论 #7628913 未加载
评论 #7628934 未加载
评论 #7629475 未加载
评论 #7632260 未加载
评论 #7628923 未加载
sandGorgon大约 11 年前
<i>We rewrote Twisted Deferred handling to be 30% faster. We optimized the Twisted Reactor event loop to not only do less constant work, but to prefer incoming I&#x2F;O.<p>...We spent long nights poring over profiling readouts and traces, wrote our own monitoring system, and wrote our own benchmarking tools<p>... We enabled cgroups on our Python process to tightly control memory usage and swap contention without requiring a different Python memory allocator scheme. </i><p>This is very interesting for the rest of the Python&#x2F;Twisted community - now that they are not using this (and so is&#x27;nt part of the secret sauce), I wonder if they are willing to push this out as a patch&#x2F;blog post ?
评论 #7628760 未加载
sigil大约 11 年前
It sounds like the cost of context switching between these very different workloads (crypto, disk I&#x2F;O, network I&#x2F;O, protocol buffer parsing and formatting) could be improved in Twisted.<p>Any idea where the overhead comes from? Twisted, or the Python interpreter itself? Is this a GIL performance issue? Or perhaps even lower -- something here is really hostile to CPU cache?<p>I realize this is a matter of taste, but my favorite async framework is still the kernel. Write small programs that do one thing well (and thus have pretty uniform workloads) and then let the kernel balance resource usage.<p><i>After continuing to hit walls with the standard Python Protocol Buffer library, we wrote our own that was 5x faster than the barely-documented C++ Python Protocol Buffer compiled module support.</i><p>Ugh yeah, the standard Python protobuf library is pure python and horribly slow. And it requires code generation -- in a dynamic language! The C++ one is faster, but also requires code generation and is just nasty to work with.<p>Not that this matters much to you at this point, but I have a small C&#x2F;Python protobuf library that&#x27;s 10x faster than the standard Python protobuf: <a href="https://github.com/acg/lwpb" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;acg&#x2F;lwpb</a><p>PS. I see you&#x27;re in SLC area -- me too. We should talk tech shop in person sometime!
评论 #7629002 未加载
cgag大约 11 年前
I like seeing people embrace static typing, but it makes me sad that it&#x27;s a language where you can&#x27;t even write a container (a map&#x2F;dictionry&#x2F;linked-list&#x2F;etc) without giving up type safety.<p>I recognize that Golang is filling a gap, letting people get static typing without having to learn much new stuff, and there&#x27;s value in that, but I think Go enthusiasts should read these great posts by Tikhon Jelvis with an open mind:<p><a href="https://www.quora.com/Googles-programming-language-Go-seems-to-be-the-most-modern-and-well-featured-language-in-existence-today-Is-that-the-case-or-are-there-major-drawbacks-that-Im-not-seeing" rel="nofollow">https:&#x2F;&#x2F;www.quora.com&#x2F;Googles-programming-language-Go-seems-...</a><p><a href="https://www.quora.com/Go-programming-language/Do-you-feel-that-golang-is-ugly" rel="nofollow">https:&#x2F;&#x2F;www.quora.com&#x2F;Go-programming-language&#x2F;Do-you-feel-th...</a>
评论 #7630917 未加载
评论 #7632780 未加载
wting大约 11 年前
I think people are rediscovering the benefits of static typing for performance and correctness reasons. The latest generation of statically typed languages[0] all have some level of type inferencing which helps.<p>If Python is fast enough for you, it’s a fantastic language. The problem is once performance or codebase demands scale, dynamic typing rears its ugly head and there are no simple solutions. At work we sidestep this issue by writing a plethora of tests, but now dynamic typing productivity gains are offset and we spin up a lot of AWS instances for performance.<p>[0] Go, Rust, Scala. Haskell and OCaml have had it for a while.
评论 #7629598 未加载
评论 #7629512 未加载
hyp0大约 11 年前
Wouldn&#x27;t java be the natural rewrite, for performance speed up? Java is betwen Python and C, in terms of ease-of-use and performance. They mention a C rewrite, but not a java one...<p>I guess the concept here is that Go is between Python and Java: (much of) the ease of use of Python + (much of) the performance of Java. The type system literally straddles the dynamic and static.<p>[ I also detect an enthusiasm for cool, new tech for its own sake (despite risks and cost, or whether it&#x27;s actually better or not - <i>hell let&#x27;s find out!</i>) ]
评论 #7629139 未加载
评论 #7629086 未加载
评论 #7629055 未加载
评论 #7630266 未加载
melling大约 11 年前
&quot;So in the bottom of the ninth, we decided to transliterate our 90k lines of Python directly to Go, line by line.<p>It took us about 4 weeks.&quot;<p>How many people worked on the project? What&#x27;s the build time on a program of this size?
评论 #7628757 未加载
评论 #7628761 未加载
TD-Linux大约 11 年前
I wrote some Python to run on an OpenWRT (MIPS-based, 400MHz) router that interfaced with an Xbee radio and ran a web interface. I quickly discovered that what I expected to be IO-bound was in fact CPU-bound - reformatting the data and all the related text operations were fast enough after optimization, but only by a hair.<p>Our solution was to switch to a dedicated ARM board (beaglebone) attached to the router. But I&#x27;m definitely going to take a look at using a compiled language now, as the codebase is still very small.
andybak大约 11 年前
I&#x27;ve got a completely data-free feeling that Ruby people hitting performance bottlenecks seem to reach for node whilst Python people have a tendency to look at Go.<p>Does anyone else think that this rings true? And if so - is it a cultural thing?
评论 #7629408 未加载
评论 #7628949 未加载
评论 #7628947 未加载
评论 #7629540 未加载
评论 #7629815 未加载
jessaustin大约 11 年前
<i>After all of this optimization, we got up to 1.2 MB&#x2F;s.</i><p>Ouch. That was a lot of work for 0.2 MB&#x2F;s. The next 2.8 MB&#x2F;s was <i>also</i> a lot of work, but it seems conceptually more straightforward.
评论 #7628692 未加载
wisty大约 11 年前
That&#x27;s actually kind of shocking. I thought it would take way more work to port from Python to Go, and a 400% speedup is way less than I&#x27;d have expected (though this is string handling, and the Python was already brutally optimised with C modules where it counted).
评论 #7628896 未加载
phaedryx大约 11 年前
I&#x27;m curious what the memory usage was like before and after.
评论 #7628688 未加载
dekhn大约 11 年前
One thing to mention, since the OP talks about developing for ARM:<p>I was surprised recently: first that Go itself bundles a cross compiler (you can easily download Go on x86 and compile ARM binaries with it), second that this made developing for Raspberry Pi and similar platforms very productive, and third that many Linux distributions have QEMU and set up binfmt translation so ARM-compiled binaries run on x86 with no extra effort.
评论 #7630781 未加载
dalek2point3大约 11 年前
Can someone tell me how much control GOOG has over the development of Go? I&#x27;m interested, but worry that is this another GOOG &quot;open source&quot; project that they have a huge say in.
评论 #7629655 未加载
评论 #7629575 未加载
impostervt大约 11 年前
Is anyone else getting SSL errors on the site?
评论 #7628866 未加载
flavoie大约 11 年前
I&#x27;m curious if they tried Cython. I&#x27;ve read that you can achieve up to 35% speedup just by compiling python code, plus you can type pyx file to get near C performance.<p>I never tried it, this would have been a nice use case.
评论 #7629092 未加载
adem大约 11 年前
&quot;So in the bottom of the ninth, we decided to transliterate our 90k lines of Python directly to Go, line by line.&quot;<p>A question here: Is it always a practical approach to transliterate line by line when transitioning to another language or because &quot;Go [is] semantically very similar to Python&quot;?
评论 #7629633 未加载
评论 #7630597 未加载
评论 #7631475 未加载
jlafon大约 11 年前
If you did a line for line transliteration, did you use any of Go&#x27;s concurrency features such goroutines or channels? I ask because I find that when I translate a program from Python to Go that it&#x27;s beneficial to structure the program differently so that I can use goroutines.
评论 #7629107 未加载
评论 #7629181 未加载
dilap大约 11 年前
I&#x27;m curious as to how they did a line-by-line port of the twisted program -- twisted&#x27;s async (plus coroutines-light w&#x2F; via generators) seems like a significantly different model than goroutines + channels. I wonder if they actually implemented Deferred in Go?
评论 #7629725 未加载
micro_cam大约 11 年前
I really hope the go compilers will soon support shared object output so we can write python extensions in go instead of having to fully dump one for the other. I know it was a maybe for go 1.3 but I haven&#x27;t heard much chatter and fear it will get pushed backed.
评论 #7629735 未加载
stuaxo大约 11 年前
Did you release your faster version of the Protocol Buffer library ?
评论 #7628798 未加载
azatris大约 11 年前
What are the disadvantages of using Go instead of Python?
评论 #7633665 未加载
bsder大约 11 年前
Why not just bump your processing power?<p>No offense, but that was a hideous amount of effort for just 20% in one area. An extra dollar or two spent on a hardware upgrade wins for everything.
azth大约 11 年前
&gt; provided that your problems are not algorithmic complexity<p>What does that even mean?
评论 #7631239 未加载
aet大约 11 年前
Can someone explain why SpaceMonkey is better than Dropbox?
评论 #7632524 未加载
azth大约 11 年前
Was the CPU pegged at 100% when using Python then?
评论 #7632098 未加载
jpatel3大约 11 年前
fyi..login link fails on the site.
leccine大约 11 年前
Well it is kind of obvious that a compiled language is going to be faster than an interpreted one, especially the way how these interpreters work (<a href="https://wiki.python.org/moin/GlobalInterpreterLock" rel="nofollow">https:&#x2F;&#x2F;wiki.python.org&#x2F;moin&#x2F;GlobalInterpreterLock</a>). You can fool yourself with Twisted (or in Ruby with EventMachine, Goliath etc.) but it gets so just a bit ahead. The surprising fact for me is that Go is not as much faster. I was expecting a bigger gap in the performance between Go and Python. Understanding where your bottlenecks are is crucial and it is not super hard in Go. <a href="https://www.datadoghq.com/2014/04/go-performance-tales/" rel="nofollow">https:&#x2F;&#x2F;www.datadoghq.com&#x2F;2014&#x2F;04&#x2F;go-performance-tales&#x2F;</a> I guess the SpaceMonkey guys might further improve the performance just by doing a thorough analysis on their code.
评论 #7628786 未加载
评论 #7628824 未加载
评论 #7628869 未加载
评论 #7629968 未加载
评论 #7630027 未加载
评论 #7629179 未加载
sfk大约 11 年前
And the website takes more than 30s to load...
评论 #7629303 未加载
评论 #7629096 未加载
评论 #7629321 未加载