TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Reducing Python's startup time

219 点作者 vanni超过 7 年前

26 条评论

raymondh超过 7 年前
To be clear. I never said that start-up time wasn&#x27;t important. Instead, I pointed out that in the previously studied example, the measured impact on start-up time of a single named tuple was a fraction of a millisecond and less than 0.1% of start-up time.<p>At Guido&#x27;s behest, I agreed to treat this miniscule component of start-up time as being important and will review and implement some optimizations in the sprints next week. I&#x27;ve agreed to do this even if it comes at the expense of code clarity, complexity, or maintainability. I expect that when I&#x27;m done, _source will be a lazily generated attribute and that much of the code currently in the template will be pre-computed. This seems reasonable to me.<p>FWIW, I&#x27;ve devoted 16 years of my life to optimizing Python and care a great deal about start-up time. I&#x27;m not happy about all the Raymond bashing going on in this thread.
评论 #15135817 未加载
评论 #15135606 未加载
评论 #15136002 未加载
评论 #15139468 未加载
NelsonMinar超过 7 年前
namedtuple is particularly weird. When you create a new type of namedtuple, Python creates the source code for a new class from a string template and passes it to exec() to build the class. It&#x27;s a clever and relatively straightforward bit of metaprogramming, but it&#x27;s not a suprise it&#x27;s not fast. <a href="https:&#x2F;&#x2F;hg.python.org&#x2F;cpython&#x2F;file&#x2F;b14308524cff&#x2F;Lib&#x2F;collections&#x2F;__init__.py#l236" rel="nofollow">https:&#x2F;&#x2F;hg.python.org&#x2F;cpython&#x2F;file&#x2F;b14308524cff&#x2F;Lib&#x2F;collecti...</a><p>(BTW if you need a quick and dirty way to make Python start up faster, see if you can use the -S flag for your application. It skips the site-specific initialization and has a significant impact on startup times.)
评论 #15134031 未加载
评论 #15134726 未加载
评论 #15133166 未加载
评论 #15132779 未加载
评论 #15135491 未加载
sevensor超过 7 年前
I&#x27;d been wondering why _source was in there. I&#x27;m a little surprised that Hettinger thinks it&#x27;s so important to retain it. It&#x27;s a neat demonstration of how to make a new data type, and I like that it&#x27;s so straightforward, but I don&#x27;t buy the argument that slowing down the interpreter is important for didactic reasons.<p>Edit: I&#x27;m talking about namedtuple, the implementation of which is slow because namedtuple() has to construct _source for every new namedtuple type. Which was clear from the article but not from my comment.
评论 #15135569 未加载
评论 #15134945 未加载
评论 #15133320 未加载
takeda超过 7 年前
Seems like this article is inflammatory and totally unnecessary.<p>It is written in a way to pin a blame on someone (Raymond H) and also looking at python-dev (and even on the github issue) the decision was made month earlier (July) to go with the optimization[1], and Raymond complied[2].<p>[1] <a href="https:&#x2F;&#x2F;mail.python.org&#x2F;pipermail&#x2F;python-dev&#x2F;2017-July&#x2F;148606.html" rel="nofollow">https:&#x2F;&#x2F;mail.python.org&#x2F;pipermail&#x2F;python-dev&#x2F;2017-July&#x2F;14860...</a> [2] <a href="https:&#x2F;&#x2F;mail.python.org&#x2F;pipermail&#x2F;python-dev&#x2F;2017-July&#x2F;148611.html" rel="nofollow">https:&#x2F;&#x2F;mail.python.org&#x2F;pipermail&#x2F;python-dev&#x2F;2017-July&#x2F;14861...</a>
eesmith超过 7 年前
I have grown to seriously dislike &#x27;namedtuple&#x27;. It seems great, and it&#x27;s so easy to use, but there&#x27;s so few places it can be used without dragging in future backwards-compatibility baggage.<p>Let&#x27;s say you have a light-weight object, like a &quot;full name&quot; which can be represented with a &quot;forename&quot; and an &quot;surname&quot;. (Yes, there are many names which don&#x27;t fit this category. This is a toy example.) So you write:<p><pre><code> from collections import namedtuple FullName = namedtuple(&quot;FullName&quot;, &quot;forename surname&quot;) name = FullName(&quot;Edward&quot;, &quot;Smith&quot;) </code></pre> So simple! So clean! No &quot;self.forename = forename&quot; or need to make your own __repr__.<p>Except you also inherit the tuple behavior, because this isn&#x27;t really light-weight object but a ... let&#x27;s call it a heavy-weight tuple.<p><pre><code> &gt;&gt;&gt; name[0] &#x27;Edward&#x27; &gt;&gt;&gt; name[1] &#x27;Smith&#x27; </code></pre> This mean people may (okay, are likely) do things like:<p><pre><code> forename, surname = name </code></pre> or even<p><pre><code> for forename, surname in names: ... </code></pre> Even if that&#x27;s not what you want in your API. If you care about backwards compatibility, then if you replace the namedtuple with its own class then you&#x27;ll have to re-implement __getitem__, _replace, __len__, and count - methods which make no sense for a simple name class.<p>Or, if you decided to change the API, to make it be:<p><pre><code> FullName = namedtuple(&quot;FullName&quot;, &quot;forename middlename surname suffix&quot;) </code></pre> then you&#x27;ll again break code which used the full indexing API that you originally implicitly promised, simply by saying this is a namedtuple.<p>&#x27;namedtuple&#x27; has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. For example, the old tuples once returned from os.stat() and time.gmtime().<p>It&#x27;s also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use like destructuring assignment, or can update all uses if you want to change the API. For example, a light-weight object used in a test-suite or internal helper function.<p>Otherwise, don&#x27;t use it. Spend the extra few minutes so in the future you don&#x27;t have to worry about backwards compatibility issues.
评论 #15132789 未加载
评论 #15133082 未加载
评论 #15132972 未加载
评论 #15133621 未加载
评论 #15133008 未加载
评论 #15133055 未加载
评论 #15135440 未加载
评论 #15133999 未加载
评论 #15134760 未加载
makecheck超过 7 年前
Moving &quot;import&quot; statements into dynamically-controlled blocks goes a long way in my experience, despite being flagged by tools like &quot;pylint&quot;.<p>Buried imports free the interpreter from doing something until it is actually required; really nice if you just want to run &quot;--help&quot; and not wait 4 seconds for utterly unrelated modules to be found. It also creates this interesting situation where a script can technically be broken (e.g. module not found) but you don&#x27;t <i>care</i> as long as the <i>part</i> you&#x27;re using is OK.<p>Grouped imports are undoubtedly nice for purity and easily seeing dependencies but they may not be smart in a dynamic language. It is still pretty easy to &quot;grep&quot; to find imports if you&#x27;re trying to track dependencies.
评论 #15137179 未加载
Boxxed超过 7 年前
One side benefit to removing the use of &quot;eval&quot; is that namedtuples will then be pickle-able. I&#x27;m surprised this hasn&#x27;t come up: every now and then I hit some weird data I want to quick-and-dirty serialize but can&#x27;t because there&#x27;s a namedtuple buried deep inside it.
评论 #15137245 未加载
评论 #15137144 未加载
mikepurvis超过 7 年前
One of the comments below the article mentions this, but in my experience on Linux and Mac, by far the biggest culprit is pkg_resources and its nasty habit of spidering the filesystem:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;setuptools&#x2F;issues&#x2F;926" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;setuptools&#x2F;issues&#x2F;926</a><p>There are hacks to get around it, but it&#x27;s a deep hole.
评论 #15132438 未加载
jordigh超过 7 年前
Mercurial did a lot of work to reduce startup time via its demandimport module.<p><a href="https:&#x2F;&#x2F;www.mercurial-scm.org&#x2F;repo&#x2F;hg&#x2F;file&#x2F;tip&#x2F;hgdemandimport&#x2F;demandimportpy2.py#l1" rel="nofollow">https:&#x2F;&#x2F;www.mercurial-scm.org&#x2F;repo&#x2F;hg&#x2F;file&#x2F;tip&#x2F;hgdemandimpor...</a><p>Basically, it&#x27;s a lazy loading of all imports. You can write `import foo` but it won&#x27;t actually be imported until you do `foo.whatever()`.<p>It&#x27;s a crutch, and it&#x27;s true that hg still pays the overall startup cost of Python. However, even with &quot;45 times slower than git&quot;, the situation is not so dire:<p><pre><code> jordi@chloe:~$ time hg --version Mercurial Distributed SCM (version 4.2.2) (see https:&#x2F;&#x2F;mercurial-scm.org for more information) Copyright (C) 2005-2017 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. real 0m0.096s user 0m0.084s sys 0m0.008s jordi@chloe:~$ time git --version git version 2.1.4 real 0m0.001s user 0m0.000s sys 0m0.000s </code></pre> I really can barely perceive a difference between 0.096s and 0.001s. Since it really is a one-time startup cost, it&#x27;s not like we can even say that this difference accumulates and that hg is overall 45 times slower than git.<p>Pierre-Yves also has an interesting talk about all of the tricks that have to go into hg in order to make it fast with Python. There&#x27;s stuff like doing `method = object.method; method()` instead of doing `object.method()` over and over again to avoid paying the cost of method lookup and so forth:<p><a href="https:&#x2F;&#x2F;2015.pycon.ca&#x2F;en&#x2F;schedule&#x2F;53&#x2F;" rel="nofollow">https:&#x2F;&#x2F;2015.pycon.ca&#x2F;en&#x2F;schedule&#x2F;53&#x2F;</a>
评论 #15133161 未加载
评论 #15133853 未加载
wyldfire超过 7 年前
BDFL says:<p>&gt; Concluding, I think we should move on from the original implementation and optimize the heck out of namedtuple.<p>I remember reading this on the mailing list and thinking &quot;yes, namedtuple deserves optimization because it&#x27;s an excellent resource&quot;.<p>For my use cases, I don&#x27;t think I notice Python&#x27;s startup time. When I write Python code it&#x27;s usually not performance critical. When I do write performance-critical Python code, I usually care about total runtime and PyPy is usually a win here. Aside: if you use PyPy, you should probably be using namedtuples. IIRC it models those much better than it does dicts. And IMO namedtuples model many common data structures better than dicts do.<p>Just because I don&#x27;t feel the pain of Python&#x27;s startup time doesn&#x27;t mean we shouldn&#x27;t try to optimize it. I think I find my way into some esoteric Python&#x2F;CPython corners but FWIW I&#x27;ve never needed namedtuple&#x27;s &quot;_source&quot; attr.
评论 #15132913 未加载
Redoubts超过 7 年前
&gt; The approach of generating source code and exec()ing it, is a cool demonstration of Python&#x27;s expressive power<p>Man, I don&#x27;t see many positive opinions of exec out there.<p>&gt; I think we should move on from the original implementation and optimize the heck out of namedtuple. The original has served us well. The world is constantly changing. Python should adapt to the (happy) fact that it&#x27;s being used for systems larger than any of us could imagine 15 years ago.<p>I really wish the BDFL could say the same for CPython itself. This continued insistence of treating python as a teaching language with a teaching implementation is really weird.
jxcl超过 7 年前
Raymond Hettinger, if I remember correctly, is the person who originally wrote NamedTuple, and he&#x27;s not shy about reminding people during his talks at conferences. I wonder if his pride or other personal feelings about that particular module could be clouding his judgment over whether or not to optimize it?
评论 #15135804 未加载
评论 #15132857 未加载
jorams超过 7 年前
&gt; The approach of generating source code and exec()ing it, is a cool demonstration of Python&#x27;s expressive power<p>This sentence is very surprising to me. If the best way to implement something is to literally build a <i>string of source code</i> and pass it to the interpreter, to me it means the user is unable to really express what they want in the language.<p>As a (terrible) example, imagine a language whose number addition operator only works on literals.<p><pre><code> 1 + 2 =&gt; 3 a = 1 b = 2 a + b =&gt; ERROR! </code></pre> It does, however, have a string concatenation operator that also works on variables and a way to read user-provided numbers into a variable, as well as some kind of eval function. What would you do to add two user-provided numbers? Something like the following:<p><pre><code> foo = read number! bar = read number! baz = concatenate x, &#x27; + &#x27;, y quux = eval baz </code></pre> The same thing could be stated about this language...<p>&gt; The approach of generating source code and eval-ing it, is a cool demonstration of &lt;this language&gt;&#x27;s expressive power.<p>...but it clearly wouldn&#x27;t make sense. Functions like exec and eval are an escape hatch for when there is no sensible way of expressing something in the language.
评论 #15137238 未加载
评论 #15134454 未加载
epberry超过 7 年前
Ah, slow startups on unittests is a big headache. This is especially apparent on big projects like Django where you might as well leave your desk and grab coffee when the unittests start, even for small projects. That said, I can&#x27;t help but think that this is just an inevitable tradeoff of Python. You want great expressive power? You have to sacrifice speed. Awesome constructs like namedtuple make this worth it imo.
评论 #15135065 未加载
评论 #15133046 未加载
chubot超过 7 年前
I agree with optimizing Python&#x27;s startup time, and I agree that namedtuple is weird and should be changed.<p>But I doubt that the namedtuple change will noticeably decrease startup time for most applications (after having experience with this problem for 10+ years).<p>From a comment over 3 years ago:<p><i>[Python interpreter startup does] random access I&#x2F;O (the slowest thing your computer can do) proportional to (large constant factor) x (num imports in program) x (length of PYTHONPATH).</i><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7842873" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7842873</a><p>I don&#x27;t think that instantiating a Python parser 100 times for exec() is going to compare to that. I guess the difference is that I&#x27;m thinking about the cold cache case -- namedtuple might be noticeable in the warm cache case.<p>And there are many many command line tools that start slow because they&#x27;re written in Python, not just Mercurial.<p>Mercurial is actually one of the best because they care about it (demandimport) and they don&#x27;t have too many dependencies. IIRC some of the Google Cloud tools take 500-1000+ ms to start because they are crawling tons of dependencies with long PYTHONPATH.
评论 #15134149 未加载
评论 #15133329 未加载
Walkman超过 7 年前
At my company we changed the interaction between two daemons by calling a command line script instead. It caused a 300ms delay in every single AJAX call and broke a dozens of end-to-end tests.<p>I just tried with Python 2.7 and 3.6 and the hello world startup time is 100ms which is awful lot.
evmar超过 7 年前
I suspect one factor for why Go is so popular within Google is that (due to complex path hacks as mentioned here as well as a network file system) Python startup is so slow.<p>On an app I worked on the unit tests took &gt;10 seconds to boot on each run. Among the languages available to Google developers, Go ends up being the lightest weight, despite there being plenty of other lighter languages available outside. (This is not the place to start an argument about language plurality.)
hathawsh超过 7 年前
I feel like the namedtuple issue could be solved cleanly by adding a feature I&#x27;ve wanted for a long time in Python: some simple way to cache high-level data and code in generated .pyc files. Today, there&#x27;s no way to do that; Python compiles the code but evaluates nothing until execution time. I&#x27;d like my .pyc files to contain some precomputed expressions and code, to reduce startup time. I probably ought to discuss this on python-ideas.
fractalb超过 7 年前
If the namedtuple is used so heavily, why not make it a part of built-in functions?
ericfrederich超过 7 年前
It&#x27;s funny. Go on #python on freenode and ask a question about named tuple. Consensus there is that they should be avoided in favor of real classes or attrs decorated classes.
first_amendment超过 7 年前
It&#x27;s unfortunate that the new version still creates __new__ using exec(). Doesn&#x27;t seem necessary at all. Instead of generating the method as a string with the argument names filled in, why not use use a combination of * n and * * kw?
est超过 7 年前
IMHO this namedtuple is non-issue compared to buildout.<p>buildout tends to append millions of path to os.path<p>Annnnd your python project start time requires 1 minute or more because each import requires scan millions of directories.
camus2超过 7 年前
My question is, is it worse or better with Ruby,NodeJS, Perl,Java or Go? I remember NodeJS scripts being extremely slow to start in general, way slower than Python.
评论 #15133903 未加载
wiradikusuma超过 7 年前
heh, when I read it, I thought it was about reducing <i>Java</i> startup time to be on par with Python&#x27;s.
NewEntryHN超过 7 年前
<p><pre><code> $ time python -c &#x27;&#x27; real 0m0.071s user 0m0.064s sys 0m0.000s $ time python -c &#x27;import functools&#x27; real 0m0.073s user 0m0.052s sys 0m0.016s </code></pre> I mean...
评论 #15135431 未加载
sillysaurus3超过 7 年前
<p><pre><code> $ time python -c &#x27;print(1)&#x27; 1 real 0m0.067s user 0m0.015s sys 0m0.025s </code></pre> Is it slow? That was Python 2.7.10, and for me Python 3.6 is similarly fast.
评论 #15132387 未加载
评论 #15132358 未加载
评论 #15132594 未加载
评论 #15133227 未加载
评论 #15134430 未加载
评论 #15132380 未加载