This is an excessively long blog post that basically states: do stream processing when your data set doesn't fit into memory.<p><pre><code> with open('in.txt') as input, open('out.txt') as out:
for line in input.readlines():
out.write(foo(line))
</code></pre>
Python users are used to importing everything all at once, while in C everything is done in small chunks whenever possible.<p>Python 3 is also moving into this direction by replacing many default functions with their iterator equivalents (map, range, etc).<p>You might think that this means forcing everything into one big context manager, but that's not necessarily true. For example:<p><pre><code> from itertools import imap
def read_file(filename):
with open(filename, 'r') as f:
reader = csv.reader(f)
for line in reader:
yield line
def write_file(filename, data):
with open(filename, 'w') as f:
writer = csv.writer(f)
map(writer.writerow, data)
write_file(
filename='out.txt',
data=imap(foo, read_file('in.txt')))</code></pre>