TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

An introduction to parallel programming using Python's multiprocessing module

104 pointsby rasbtalmost 11 years ago

10 comments

platzalmost 11 years ago
I feel that a discussion of <a href="http://en.wikipedia.org/wiki/Amdahl&#x27;s_law" rel="nofollow">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Amdahl&#x27;s_law</a> should be mandatory when introducing parallel programming<p>Small sequential portions of an otherwise parallel algorithm can have huge effects on the overall running when trying to scale up.<p>&quot;parconc&quot; explains this while discussing a parallel version of k-means, talks about how things like granularity of data needs to be fine-tuned for parallel algos, and provides some nice visualizations into what the CPU&#x27;s are actually doing on a timeline: <a href="http://chimera.labs.oreilly.com/books/1230000000929/ch03.html#sec_par-kmeans-perf" rel="nofollow">http:&#x2F;&#x2F;chimera.labs.oreilly.com&#x2F;books&#x2F;1230000000929&#x2F;ch03.htm...</a><p>Overall I think multicore is a good tool to have in your toolbox, but it seems like there needs to be a lot of tuning and effort to get good rewards for the time invested.
评论 #8026073 未加载
d0vsalmost 11 years ago
<a href="https://docs.python.org/dev/library/concurrent.futures.html" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;dev&#x2F;library&#x2F;concurrent.futures.html</a>
abemassryalmost 11 years ago
I wrote <a href="https://github.com/abemassry/crazip" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;abemassry&#x2F;crazip</a> in python and it was challenging to find out how to do multiprocessing effectively, not sure how this would run if implemented in other laungaues.
yohanatanalmost 11 years ago
This article contains a perfect example of why I don&#x27;t like to write or read comments in code. Comments are not compiled and thus allow for sloppy verbiage such as the following:<p><pre><code> # Exit the completed processes for p in processes: p.join() </code></pre> The comment should read something more like: &quot;Wait for all the subprocesses to exit&quot; but is that really any more helpful than just reading the code and seeing that join is called on each subprocess and connecting the dots from there?
评论 #8026670 未加载
huevingalmost 11 years ago
Instead of calling the &#x27;else&#x27; condition of the for loop a &#x27;completion-else&#x27;, just call it the &#x27;nobreak&#x27; condition. Unlike &#x27;completion-else&#x27;, &#x27;nobreak&#x27; immediately describes when it will be executed.
webmavenalmost 11 years ago
Better starting point: <a href="https://medium.com/@thechriskiehl/parallelism-in-one-line-40e9b2b36148" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;@thechriskiehl&#x2F;parallelism-in-one-line-40...</a>
huevingalmost 11 years ago
&gt; x_i = (point_x - row[:,np.newaxis]) &#x2F; (h)<p>TypeError: list indices must be integers, not tuple
评论 #8027190 未加载
gomesnayagamalmost 11 years ago
Still multiprocessing is experimental phase for various use case :(
thikonomalmost 11 years ago
pretty poor language choice to teach parallel programming concepts
zobzualmost 11 years ago
the author seems to be unaware of threading and event based models.<p>multiprocessing adds memory isolation through the CPU&#x27;s protected memory.
评论 #8026045 未加载
评论 #8026220 未加载
评论 #8026026 未加载