Ask HN: Are my HPC professors right? Is Python worthless compared to C?

&gt; This code has to run in parallel, and run fast<p>Then you&#x27;re certainly not going to get away with writing it all in Python, but it&#x27;s a very common paradigm to write hotter parts of the code in faster languages and then glue everything together in Python. I don&#x27;t see why that wouldn&#x27;t work here.<p>&gt; My thought is to write it in Python, profile it, and if needed, rewrite the slow parts in C<p>That&#x27;s a very reasonable and common approach if you aren&#x27;t already confident in which parts will need the extra performance ahead of time.<p>&gt; Should a young PhD student in HPC really be investing all their time in C and not consider Python as a reasonable solution?<p>You should absolutely be using both together, each to their respective strengths. The only thing unreasonable about any of this is the idea of pitting the languages against each other and acting like one needs to win.

I&#x27;m an old PhD student. I&#x27;ve seen cases where an easy &#x2F; naive solution written in Python can be orders-of-magnitude slower than a solution written in C, but I suspect that you&#x27;re right: that a thoughtful &#x2F; clever use of Python should be perfectly fine. I&#x27;ve also seen that professors don&#x27;t know everything, and what they do know can be dated.<p>More importantly, though, I think a young PhD student should not pick a fight with his advisor and committee members or try to prove them wrong. Generally, do what they suggest and give them credit for it, or at least thank them enthusiastically for the suggestions and don&#x27;t make a big deal about not following them. You&#x27;re at their mercy for getting the PhD, and it&#x27;s subjective, and their opinion of <i>you</i> probably matters at least as much as their opinion of <i>your work.</i> This is one the things that I learned from ~15 years of professional work under many different bosses before starting my PhD program, and something that I think many young students still need to learn.

I did my PhD after 20+ years in industry, lots of it programming in C on AIX&#x2F;HPUX&#x2F;Solaris systems. I ended up learning Python to complete my research work.<p>Granted compared with Python, C is verbose and the edit-compile-debug-run is a drag. However, C as a language is not too bad. It is the libraries and APIs that slow me down. Often times the abstractions are too leaky or a poor fit for what needs to be done.<p>What works for me is to test and refine algorithms in Python. When it works well, then I use the Python code as <i>pseudo-code</i> and translate to even more optimized C code. It helps to modularize your Python code, so that you only need to port the performance critical portions to C and the rest can remain in Python.<p>Of course, you still need to learn and gain experience with C. Personally I wouldn&#x27;t put much faith in Python to C transpilers. For optimal performance, you really need to understand your algorithms and data structuring. These days understanding how caching and locality of code and data impacts performance is crucial to writing performant code.<p>BTW have you considered using CUDA, etc for your finite element code? GPGPUs are ideal for that sort of computation. Lots of potential for parallelization.

Few things are black and white to the point they should be saying a tool is worthless. Nor should we be declaring a binary right&#x2F;wrong on their take on it. There is nuance and grey areas to all things, so the question is less &quot;Are they right?&quot;, and more &quot;What drives them to say such a thing?&quot;

If you&#x27;re studying HPC, I would say that it&#x27;s probably worth learning how to do it the hard way, especially if you will have other projects that require doing so.<p>However, if you want to use Python, I would consider JAX. It has a nice set of operations based on numpy and scipy and can compile to many backends (GPU, TPU, CPU, etc.) using XLA. The compiler is great at finding ways to optimize routines that you might not think of. Some of the manual parallelism functionality is still considered experimental, but I haven&#x27;t seen that cause any issues or prevent functionality.

Other commenters gave excellent and actionable answers to your question. I want to quibble about your reaction to C.<p>&gt; I do not enjoy working with C at all. Its esoteric and difficult to understand, and just overall feels like I&#x27;m using a tool from the 70s.<p><i>Esoteric</i> means &quot;intended for or likely to be understood by only a small number of people with a specialized knowledge or interest.&quot; That does not describe C, a language widely understood and used by a large number of programmers, across application domains and programming interests.<p><i>Difficult to understand</i> describes a reaction you have to learning C, not a property of the language. Again a very large number of programmers understand and use C, have for decades, and a huge amount of C code gets written and maintained constantly. The C language includes very few keywords and a simple syntax, and a small standard library compared to Python. People new to C usually trip over memory management, pointers, and the overall philosophy behind C, not learning the language itself.<p>C does date back to the late 1970s, but so does most hardware and software technology we use today. Newer does not equal better, and C has remained relevant and popular despite its age because it works so well. Toyota introduced the Corolla in the mid-1960s and it remains relevant and widely-used today, not to mention influential in the automobile industry. C occupies a similar position, a language that works so well it has staying power and has undergone relatively minor updates over time, unless you count derivative languages that expand on and perhaps improve on C -- C++, Go, Rust, Zig, many others.<p>Good luck with your project.

If you&#x27;re going the pure Python route, don&#x27;t forget to try PyPy[1], an alternative JITed implementation of the language. A seriously underrated project, IMHO. Most time it speeds up execution by a factor of 2x-4x, but improvements of about two orders of magnitude are not unheard of. See for example [2].<p>Numeric, long-running code shoud suit PyPy optimizations well.<p>[1] <a href="https:&#x2F;&#x2F;pypy.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypy.org&#x2F;</a><p>[2] <a href="https:&#x2F;&#x2F;morepypy.blogspot.com&#x2F;2017&#x2F;10&#x2F;how-to-make-your-code-80-times-faster.html" rel="nofollow">https:&#x2F;&#x2F;morepypy.blogspot.com&#x2F;2017&#x2F;10&#x2F;how-to-make-your-code-...</a>

Python will be much slower than C, but if you follow the path that you suggest and use Python mainly as glue to stick C-based performance critical parts together you could be fine.<p>I used to edit an HPC trade rag, and I&#x27;ve written a lot of performance-critical code in C and C++, and even in Java eg for high-speed trading.<p>As a now-old fresh PhD student I think that your profs are probably wrong!

&quot; My thought is to write it in Python, profile it, and if needed, rewrite the slow parts in C. &quot;<p>Your thinking is correct though sometimes, even with the best intentions, people stop at the first step. Fixing code that already works is usually not a high priority task.

C++ seems like a good middle ground. Supports more modern features than pure C, but is compiled and is faster than Python. Also it sounds like your professors support that, vs not supporting Python.

You&#x27;re in academia. The goal isn&#x27;t to write fast code it is to create (and publish!) knowledge. If it is easier for you to create knowledge in Python than in C you should use that. In HPC exact performance on a specific device with a specific runtime is uninteresting. What is interesting is how well the solution scales, what its bottle-necks and constraints are.

There are already Python wrapper libraries for PETSc: <a href="https:&#x2F;&#x2F;petsc.org&#x2F;release&#x2F;petsc4py&#x2F;" rel="nofollow">https:&#x2F;&#x2F;petsc.org&#x2F;release&#x2F;petsc4py&#x2F;</a><p>I am not a HPC user or developer, but I have written Python wrapper libraries for C libraries. It is fairly easy to do, but it looks like some of what you are looking for is already done.

This:<p><i>&gt; I want to rewrite my simulation in Python.</i><p>Is completely different than this:<p><i>&gt; My thought is to write it in Python, profile it, and if needed, rewrite the slow parts in C.</i><p>Your professors are right if you mentioned the former only and didn&#x27;t clarify to them the latter.

Using Python for glue code and compiled native code (whether C or C++ or Rust or whatever) is a classic strategy.<p>Just profile your code with something like Scalene: <a href="https:&#x2F;&#x2F;github.com&#x2F;plasma-umass&#x2F;scalene">https:&#x2F;&#x2F;github.com&#x2F;plasma-umass&#x2F;scalene</a><p>Alternatively, you can just write it in Julia.

Fortran sir, consider Fortran. HPC simulation is exactly what its mostly used for, it runs OpenMP with so little effort, and there&#x27;s even an Nvidia Fortran optimised for writing GPU kernels.<p>Fortran is very easy to learn, runs at close to C speed for general code, and it is a very productive language. I personally loved using it.

Frankly you will learn more if you try it and fail (or maybe succeed!?) than listening to the advise of professors or random internet comments.