Measuring memory usage in Python: it’s tricky

For folks looking for a good modern profiling tool for Python codebases: my current favourite is Scalene [0], which allows profiling CPU&#x2F;GPU and memory usage in a performant way. The author gave a talk about it on last PyCon US [1].<p>[0]: <a href="https:&#x2F;&#x2F;github.com&#x2F;plasma-umass&#x2F;scalene" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;plasma-umass&#x2F;scalene</a> [1]: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nrQPqy3YY5A" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nrQPqy3YY5A</a>

Remember that a lot of software not only relies on memory that the developer explicitly allocated but also on buffers and various caches, disk cache being an example. It goes a long way - kernel might <i>refuse</i> dropping disk cache when it considers it&#x27;s too hot (ex. db machine.)<p>The question &quot;how much free memory do I have&quot; or &quot;how much memory does this program use&quot; is a very hard one, and you can&#x27;t just read the &quot;MemAvailable&quot; column from &#x2F;proc&#x2F;meminfo or RSS&#x2F;PSS&#x2F;Virt from ps&#x2F;pmap.

&gt; With this particular measurement, we’re using 3083MB, or 3.08GB<p>Don&#x27;t mix up SI and binary prefixes, kids! The listing shows 3083 MiB, which is either 3.01 GiB, or 3.23 GB. There&#x27;s no mainstream interpretation giving 3.08 of something.

I recently had an adventure with Python subprocesses and memory. This isn&#x27;t just a Python thing, but it&#x27;s more of an issue because Python lacks proper multitreading: processes don&#x27;t necessarily return memory to the OS after free(), and Python doesn&#x27;t necessarily free() unused memory after a GC run. If you&#x27;re using a process pool where some work can be memory-intensive, this can massively increase total memory use as the job bounces between subprocesses.

Good timing! I&#x27;m currently battling memory issues with a project where I&#x27;m straightening full resolution panoramas. I often reach close to my machine&#x27;s limit of 16GB.<p>I think this might help me immensely. Thank you OP.

With interpreters it&#x27;s always difficult to track resource usage. It should have full blown instrumentation w&#x2F; kernel support. I am not sure python has it.