TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Experience report on a large Python-to-Go translation

274 点作者 psxuaw超过 5 年前

18 条评论

mappu超过 5 年前
Here&#x27;s another experience report: I ported a small 1 KLOC PHP project to Go this week (in some spare time between large C++ compile times). The primary goal was to reduce the number of supported languages we use.<p>The port happened in the mechanical line-by-line way, copying each PHP file to a *.go and fixing all the syntax. The project was small enough that automation wasn&#x27;t interesting.<p>I agree with the &quot;1&#x2F;3 time spent debugging the result&quot;. Another complicated facet was the lack of insertion-order preserving maps, that PHP applications end up relying on heavily. The error&#x2F;exception impedance mismatch was not a problem in practice at all.<p>According to cloc, the original PHP project (excluding vendor) is 1.0 KLOC, the resulting Go application is 1.2 KLOC. I imagined Go would have been more verbose than this, but actually most lines remained 1:1 conversions, and the Go standard library happened to cover a lot of small utility functions that had to be separately written in PHP (e.g. for string suffix matching).<p>Another interesting point is the number of comment lines in cloc appeared to drop dramatically, since real type annotations are much less verbose than PHPDoc.
评论 #22305425 未加载
评论 #22308307 未加载
jasonpeacock超过 5 年前
I&#x27;d like to compliment the author on the quality of this post. It&#x27;s very well written, data&#x2F;example driven, fair, and educational.<p>Overall, a joy to read. Thank you!
评论 #22305471 未加载
评论 #22304961 未加载
melling超过 5 年前
Go is probably more verbose because it’s missing List comprehensions, for example.<p>It needs map, filter, reduce to reduce line count. Swift, while probably not as performant as Go, makes writing in a more Pythonic style.<p><pre><code> [1,2,3,4,5,6,7,8,9].filter {$0 % 2 == 0}.map {$0 * 2}.reduce(0, +) [&quot;550&quot;, &quot;a&quot;, &quot;6&quot;, &quot;b&quot;, &quot;42&quot;, &quot;99&quot;, &quot;100&quot;].compactMap{Int($0)}.filter {$0 &lt; 100} </code></pre> <a href="https:&#x2F;&#x2F;github.com&#x2F;melling&#x2F;SwiftCookBook&#x2F;blob&#x2F;master&#x2F;functional.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;melling&#x2F;SwiftCookBook&#x2F;blob&#x2F;master&#x2F;functio...</a>
评论 #22307461 未加载
评论 #22304949 未加载
评论 #22309355 未加载
hartzell超过 5 年前
[edit: fixed links]<p>This was discussed recently on the go-nuts mailing list: <a href="https:&#x2F;&#x2F;groups.google.com&#x2F;d&#x2F;msg&#x2F;golang-nuts&#x2F;u-L7PRa2Z-w&#x2F;kfUSx81PAAAJ" rel="nofollow">https:&#x2F;&#x2F;groups.google.com&#x2F;d&#x2F;msg&#x2F;golang-nuts&#x2F;u-L7PRa2Z-w&#x2F;kfUS...</a><p>There was also discussion around an earlier post he made about the work: <a href="https:&#x2F;&#x2F;groups.google.com&#x2F;d&#x2F;msg&#x2F;golang-nuts&#x2F;WstriKt2jTA&#x2F;lsZyX4hYAwAJ" rel="nofollow">https:&#x2F;&#x2F;groups.google.com&#x2F;d&#x2F;msg&#x2F;golang-nuts&#x2F;WstriKt2jTA&#x2F;lsZy...</a>
评论 #22309370 未加载
downerending超过 5 年前
Interesting. I&#x27;d have expected more than a 50% code expansion going to Go, maybe even 3x or 5x.<p>Similarly, he&#x27;s using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C.<p>Personally I&#x27;d be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.
评论 #22305214 未加载
评论 #22308464 未加载
评论 #22308431 未加载
outworlder超过 5 年前
&gt; The problem directed the choice of Go, not the other way around. I seriously considered OCaml or a compiled Lisp as alternatives. I concluded that in either case the semantic gap between Python and the target language was so large that translation would be impractical. Only Go offered me any practical hope.<p>I wish they expanded more on this point. Do they mean that rewriting in, say, Lisp would be longer because it wouldn&#x27;t be a &#x27;port&#x27; and more like writing a new program from scratch?<p>EDIT: Spoke too soon. Reading more carefully, I answered my own question.<p>&gt; Python reposurgeon was 14 KLOC of dense code. At that scale, any prudent person in a situation like this will perform as linear and literal a translation as possible;
评论 #22336066 未加载
评论 #22305101 未加载
DougBTX超过 5 年前
There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values.<p>The original Python code using exceptions was:<p><pre><code> sink = transform3(transform2(transform1(source))) </code></pre> Making that use error return values looks quite verbose in Go, but Rust has syntax specifically for that case, making it quite manageable:<p><pre><code> sink = transform3(transform2(transform1(source)?)?)?</code></pre>
评论 #22308409 未加载
评论 #22308157 未加载
评论 #22315098 未加载
评论 #22311582 未加载
评论 #22309546 未加载
patrec超过 5 年前
Adding lookbehinds to the regexp library is a terrible idea.<p>&gt; The regexp implementation provided by this package is guaranteed to run in time linear in the size of the input.<p>Python&#x27;s is exponential, because it inherits all the non-regular &quot;regular&quot; expression mess (such as lookbehind and backrefs) from perl.<p>One would assume esr would have marinated in unix culture for long enough to be aware of this.
评论 #22307316 未加载
评论 #22307009 未加载
cik超过 5 年前
I love reading ESR, and it&#x27;s such a pleasure to see how great his writing is in 2020. This made me hark back to The Cathedral and the Bazaar - and also echoed my usage.<p>I do however find that when you have a python project that is heavily dependent on third party libraries - these things get significantly larger and more problematic. That&#x27;s not really a commentary on Go, inasmuch that it&#x27;s a byproduct of the longevity of Python.
ageofwant超过 5 年前
I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++&#x2F;Rust&#x2F;Go and wrap it. That&#x27;s how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that&#x27;s a major reason why Python dominates today.<p>Align with the core strengths of Python&#x27;s philosophy and its toolset and get the benefit. Why fight it ?
评论 #22305946 未加载
评论 #22305470 未加载
评论 #22305951 未加载
评论 #22305250 未加载
评论 #22305901 未加载
3fe9a03ccd14ca5超过 5 年前
&gt; <i>The man barrier to translation was that, while at 14KLOC of Python reposurgeon was not especially large, the code is very dense.</i><p>Reasoning about somebody else’s dense code is probably the least favorite activities. When I hear about a language being “expressive” or having “flexible syntax” I shudder.
评论 #22306544 未加载
mistrial9超过 5 年前
significant mastery ahead!<p>This is a success story and a teaching document.<p>.. have to point to this : &quot;Now that I’ve seen Go strings… holy hell, Python 3 unicode strings sure look like a nasty botch in retrospect. &quot; (!)
评论 #22305875 未加载
xiaodai超过 5 年前
If it was too slow in Python and now moving to Go. Could there a time when there is a need to move to Rust&#x2F;C&#x2F;C++ for even faster performance? Go seems an odd choice based on performance consideration alone.
评论 #22307314 未加载
评论 #22310069 未加载
评论 #22307173 未加载
jrockway超过 5 年前
Pretty interesting. It is scary to make your &quot;learn a new language&quot; task to port 14,000 lines of code, but with that in mind, this all seems to have gone well. Some random thoughts:<p>&gt; I had to write my own set-of-int and set-of-string classes<p>map[int]struct{}, map[string]struct{}<p><pre><code> ints[42] = struct{}{} &#x2F;&#x2F; insert delete(ints, 42) &#x2F;&#x2F; delete for i := range ints { ... } &#x2F;&#x2F; iterate if _, ok := ints[42]; ok { ... } &#x2F;&#x2F; exists? </code></pre> &gt; Catchable exceptions require silly contortions<p>I am not sure why go has panic&#x2F;recover, but it&#x27;s not something to use. panic means &quot;programming error&quot;, recover means &quot;well, the code is broken but I&#x27;m just a humble generic web framework and I guess maybe the next request won&#x27;t be broken, so let&#x27;s keep running&quot;. It is absolutely not for things like &quot;timeout waiting for webserver&quot; or &quot;no rows in the database&quot; as other languages use exceptions for. For those, you return an error and either wrap it with fmt.Errorf(&quot;waiting for webserver: %w&quot;, err) or check it with errors.Is and move on. Yup, you have to remember to do that or your program will run weirdly. It&#x27;s just how it is. There is not something better that maybe with some experimentation you will figure out. You have to just do the tedious, boring, and simple thing.<p>I have used recover exactly once in my career. I wrote a program that accepted mini-programs (more like rules) via a config file that could be reloaded at runtime. We tried to prove them safe, but recover was there so that we could disable the faulty rule and keep running in the event some sort of null pointer snuck in. (I don&#x27;t think one ever did!)<p>&gt; Pass-by-reference vs. pass-by-value<p>I feel like the author wants []*Type instead of []Type here.<p>&gt; Absence of sum&#x2F;discriminated-union types<p>True. Depending on what your goals are, there are many possibilities:<p><pre><code> type IntOrString struct { i int; s string; iok, sok bool } func (i IntOrString) String() (string, error) { if i.sok { return i.s, nil } else { return &quot;&quot;, errors.New(&quot;is not a string&quot;) }} func NewInt(x int) IntOrString { return IntOrString{i: x, iok: true} } ... </code></pre> This uses more memory than interface{}, but it&#x27;s also very clear what you intend for this thing to be.<p>I will also point out that switch can bind the value for you:<p><pre><code> switch x := foo.(type) { case int: return x + 1 case string: i, err := strconv.Atoi(x) return i + 1 } </code></pre> And that you need not name types merely to call methods on them:<p><pre><code> if x, ok := foo.(interface { IntValue() int }); ok { return x.IntValue() } </code></pre> You can also go crazy like the proto compiler does for implementations of &quot;oneof&quot; and have infinite flexibility. It is not very ergonomic, but it is reliable.<p>&gt; Keyword arguments<p><pre><code> type Point struct { X, Y float64 } func EuclideanDistance(a, b Point) float64 { ... } EuclideanDistance(Point{X: 1, Y: 2}, Point{3, 4}) </code></pre> &gt; No map over slices<p>This one is like returning errors. You will press a lot of buttons on your keyboard. It is how it is.<p>I personally hate typing the average &quot;simple&quot; for loop:<p><pre><code> func fooToInterface(foos []foo) []interface{} { var result []interface{} for _, f := range foos { result = append(result, f) } return result } </code></pre> But it&#x27;s also not that hard. I used to be a Python readability reviewer at Google. I always had the hardest time reading people&#x27;s very aggressive list comprehensions. It was like they HAD to get their entire program into one line of code, or people wouldn&#x27;t think they were smart. The result was that the line became a black box; nobody would read it, it was just assumed to work.<p>I really like seeing the word &quot;for&quot; twice when you&#x27;re iterating over two things.
评论 #22308827 未加载
评论 #22306256 未加载
评论 #22305924 未加载
评论 #22309467 未加载
luord超过 5 年前
I liked this summation because the migration happened for a truly valid reason: Python <i>really</i> was a bottleneck. Not that I expected ESR to succumb to hype driven development, but it&#x27;s nice to see for sure.<p>On the article itself: I just knew that error handling would have the biggest write-up, even when the one writing was someone like ESR. Gods, the error handling in Go is odious.<p>Now my obligatory opinion: If only [insert language here, Go in my current job]&#x27;s promise of producing more maintainable code was true; the reality is that it&#x27;s just the same nigh unmaintainable hell I&#x27;ve found in nearly every other project I&#x27;ve worked on. At least Python is nice to read, even (mostly) when awfully written. Oh, how I miss it.
Insanity超过 5 年前
The missing &#x27;keyword arguments&#x27; could have been replaced with a struct passed to a function, no? Unless I&#x27;m missing something from Python, in Go you could replace this type of function:<p><pre><code> func f(x int, y int, c string) </code></pre> with something like this:<p><pre><code> type funcOptions struct { x, y int c string } func f(o funcOptions) {} f(funcOptions{x:3, y:-1, c: &quot;hello&quot;}) </code></pre> So the readability hit would have been more &#x27;minimal.
评论 #22308532 未加载
评论 #22309012 未加载
评论 #22309413 未加载
评论 #22308597 未加载
tanilama超过 5 年前
One of the better read for a long time.<p>The translation assistant you write is actually very interesting. Heavily rule based but surprising to see it actually helps at all.<p>But the scale of the project itself seems still pretty limited, reimplementation could still be an option.<p>Overall good read and interesting approach
transfire超过 5 年前
If you want a real surprise, try a rewrite in Elixir.
评论 #22312642 未加载
评论 #22309511 未加载