Codon: A high-performance Python-like compiler using LLVM

Anybody claiming it&#x27;s almost Python is kidding themselves. This compiler needs to do static type checking. This is inherently impossible in Python. Not just because of some obscure corner cases that nobody uses. It&#x27;s baked into the language itself.<p>Reality-check: Why do you think type hints and type checkers like mypy and pyright take such a long time to get going and even they are not there yet? If this was all so easy with just ignoring some obscure rarely used features then mypy would work with essentially no type annotations, all just automatic inferences. Anybody who has tried to work with type annotations in Python knows how hard this is.<p>So, those guys are quite obviously overselling their product. I can understand it, academic life is hard, and once you&#x27;ve completed your Ph.D., what can you do. You need to stand out. But these claims don&#x27;t pass the smell test, sorry.

Thanks a lot for all the comments and feedback! Wanted to add a couple points&#x2F;clarifications:<p>- Codon is a completely standalone (from CPython) compiler that was started with the goal of statically compiling as much Python code as possible, particularly for scientific computing use cases. We&#x27;re working on closing the gap further both in what we can statically compile, and by automatically falling back to CPython in cases we can&#x27;t handle. Some of the examples brought up here are actually in the process of being supported via e.g. union types, which we just added (<a href="https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;releases" rel="nofollow">https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;releases</a>).<p>- You can actually use any plain Python library in Codon (TensorFlow, matplotlib, etc.) — see <a href="https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;interoperability&#x2F;python" rel="nofollow">https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;interoperability&#x2F;python</a>. The library code will run through Python though, and won&#x27;t be compiled by Codon. (We are working on a Codon-native NumPy implementation with NumPy-specific compiler optimizations, and might do the same for other popular libraries.)<p>- We already use Codon and its compiler&#x2F;DSL framework to build quite a few high-performance scientific DSLs. For example, Seq for bioinformatics (the original motivation for Codon), and others are coming out soon.<p>Hope you&#x27;re able to give Codon a try and looking forward to further feedback and suggestions!

<i>Since Codon performs static type checking ahead of time, a few of Python&#x27;s dynamic features are disallowed. For example, monkey patching classes at runtime (although Codon supports a form of this at compile time) or adding objects of different types to a collection.</i><p>This seems like a <i>very</i> different language from Python if it won&#x27;t let you do:<p><pre><code> [1, &#x27;a string&#x27;]</code></pre>

I googled &#x27;codon and django&#x27; and unsurprisingly found a lot of bioinformatic stuff. I tried to add language and compiler to no avail. The only query that got results was codon python compiler. Overall I think it&#x27;s a name that clashes with a lot of DNA&#x2F;RNA research.<p>While searching I found a paper from 2021 about Codon [1]. The author is not in the About page of Exaloop [2] but the supervisor of that thesis is there. From the &quot;Future Work&quot; section:<p>&gt; we plan to add union types and inheritance. On the IR side [Intermediate Representation], we hope to develop additional builtin transformations and analyses, all the while expanding the reach of existing passes. As far as library support, we plan to port existing high-performance Python libraries like NumPy [...] to Codon; this will allow Codon to become a drop-in replacement for Python in many domains.<p>Maybe they already did.<p>[1] Codon: A Framework for Pythonic Domain-Specific Languages by Gabriel L. Ramirez <a href="https:&#x2F;&#x2F;dspace.mit.edu&#x2F;bitstream&#x2F;handle&#x2F;1721.1&#x2F;139336&#x2F;Ramirez-glram-meng-eecs-2021-thesis.pdf" rel="nofollow">https:&#x2F;&#x2F;dspace.mit.edu&#x2F;bitstream&#x2F;handle&#x2F;1721.1&#x2F;139336&#x2F;Ramire...</a><p>[2] <a href="https:&#x2F;&#x2F;exaloop.io&#x2F;about.html" rel="nofollow">https:&#x2F;&#x2F;exaloop.io&#x2F;about.html</a>

Very interesting--Codon can generate standalone executables, object files, and LLVM IR [1]. It has strong typing for functions and argument return values [2]. Syntax looks more compact than Cython.<p>Looking forward to giving Codon a try!<p>[1]: <a href="https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;intro" rel="nofollow">https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;intro</a><p>[2]: <a href="https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;language&#x2F;functions" rel="nofollow">https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;language&#x2F;functions</a>

Unfortunately stuff like this never makes it to the upstream. And i am afraid to ask why. We had pypy for years, but never got merged with python. That is why there are still minor incompatibilities between pypy and &quot;The Python&quot;, so it&#x27;s not that useful as it might have been if it got merged with cpython at some point.

power of Python is in ecosystem of libraries, not only Python syntax.<p>Without the ecosystem of libraries, I am afraid use cases for Codon will be very very limited. Because Python developers (just like Node) got used to thinking: Need to do X? Lets see if I can pip install library that does it.<p>Ultimately, python is like super flexible glue between ecosystem of libraries that lets anyone build and prototype high quality software very quickly

Just out of curiosity: Why is it possible to compile Common Lisp Code (or Scheme, or Clojure) to high-performance native or jit-compiled code, but not Python? It is said that &quot;Python is too dynamic&quot;, but is not everything in Lisp dynamic, too?<p>And none of these languages is less powerful than Lisp, lack Unicode support, or whatever, so this can&#x27;t be the reason.

Who would create a language that only has ASCII strings in this day and age?

after reading it&#x27;s not a python compiler but a compiled language based on the python syntax

Can we change the title to say Python-like or something similar? Based on the comments so far, it seems that the detail that it compiles its own Python-inspired language, not actual Python, is lost on many.<p>EDIT: A list of differences here: <a href="https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;differences" rel="nofollow">https:&#x2F;&#x2F;docs.exaloop.io&#x2F;codon&#x2F;general&#x2F;differences</a><p>The summary minimizes with &quot;many Python programs will work with few if any modifications&quot;, but it actually looks like a substantially different language.

What&#x27;s the difference with mypyc [0] ? It also compiles Python to native code.<p>[0]: <a href="https:&#x2F;&#x2F;github.com&#x2F;mypyc&#x2F;mypyc" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mypyc&#x2F;mypyc</a>

What is the difference between Codon and Pypy other than Codon not being targeted as a drop-in replacement for Cpython?

Do people not know about numba which unlike this project is FOSS and integrates with numpy???

This gives the same feeling as AssemblyScript: it says it is one language, up to the point it isn&#x27;t. That may make it easier for some people, but feels so uncertain. Both have a very slim set of articles in place of a proper manual; they lean on their parent languages.

<i>Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead</i><p>Further down:<p><i>Codon is a Python-compatible language, and many Python programs will work with few if any modifications:</i>

&gt; &quot;...typically on par with (and sometimes better than) that of C&#x2F;C++&quot;<p>What makes it faster than C++?<p>I see this in the documentation but I am not sure it helps me (not an expert):<p>&gt; C++? Codon often generates the same code as an equivalent C or C++ program. Codon can sometimes generate better code than C&#x2F;C++ compilers for a variety of reasons, such as better container implementations, the fact that Codon does not use object files and inlines all library code, or Codon-specific compiler optimizations that are not performed with C or C++.

Ugly, confusing naming choices: ``@par`` instead of ``@parallel``.

would love to see actual benchmarks

Looks like what taichi(<a href="https:&#x2F;&#x2F;github.com&#x2F;taichi-dev&#x2F;taichi" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;taichi-dev&#x2F;taichi</a>) is doing, does this support CUDA yet?<p>additionally how does it compare to numba the compiler for python?<p>looks like python&#x27;s performance on ML and AI field will only get stronger.

Things like this are always going to be another point of failure when trying to get something to work. Now when your python code crashes, there&#x27;s a new reason something could be going wrong, in addition to the countless other reasons.

The number one question for me would be, is it interoperable with existing Python and libraries?

How does this relate to<p><a href="https:&#x2F;&#x2F;cython.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;cython.org&#x2F;</a><p>?<p>Would it be possible to write performance-sensitive parts of a Python system in Codon and link that to a CPython or PyPy runtime that supports more dynamic features?

Seems there is not bytearray implemented, can&#x27;t test further :(

Free for non-production use... it&#x27;s a &quot;no&quot; for me.

I want the opposite. Is there a project that compiles to python (either source or bytecode)?<p>Sort of a graalvm for python?

does this support any form of FFI? It&#x27;d be nice if users could shim in lightweight APIs that clone libraries like numpy&#x2F;pytorch -- it&#x27;d immediately make some machine learning super portable!

Please note, codon is not opensource. It is business source license.

Why not contribution to PyPy and Why not MyPyC

Any benchmark comparisons to mypyc yet?

Can it run PyTorch, TF etc?

&gt;&quot;Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon&#x27;s performance is typically on par with (and sometimes better than) that of C&#x2F;C++&quot;<p>Nice! A super-fast compiler LLVM compiler for Python! Well done!<p>You know, if Python is one of the world&#x27;s most popular languages, and it was originally implemented as a dynamic and interpreted language (but fast compilers can be written for it, as evinced by Codon!) -- then maybe it would make sense to take languages that were implemented as compilers -- and re-implement them as dynamic interpreted languages!<p>Oh sure -- that would slow them down by 10x to 100x!<p>But, even though that would be the case -- the dynamic interpreted versions of the previous compiled-only language -- might be a whole lot more beginner friendly!<p>In other words, typically in dynamic interpreted languages -- a beginner can use a REPL loop or other device -- to make realtime changes to a program as it is running -- something that is usually impossible with a compiled language...<p>The possibilities for easy logging, debugging, and introspection of a program -- are typically greater&#x2F;easier -- in interpreted dynamic languages...<p>Oh sure, someone can do all of those things in compiled languages too -- but typically the additional set-up to accomplish them is more involved and nuanced -- beginners typically can&#x27;t do those things easily!<p>So, I think when I think about programming languages from this point forward -- I&#x27;m going to think about them as having &quot;two halves&quot;:<p><i>One half which is a compiled version.</i><p><i>And another half -- which is a dynamic interpreted version...</i><p>Usually when a new programming language is created in world, it is created either as a compiled language <i>or</i> as a dynamic interpreted language -- <i>but never both at the same time!</i><p>Usually it takes the work of a third party to port a given language from one domain to the other, usually from dynamic interpreted to compiled, but sometimes (as is sometimes the case with scripting languages derived from compiled languages), sometimes in the reverse!<p>Point is: <i>There are benefits to be derived from each paradigm, both dynamic interpreted and compiled!</i><p>So why do we currently look at&#x2F;think about -- most computer languages -- as either one or the other?<p>I&#x27;m going to be looking at all computer languages as potentially <i>both</i>, from this point forward...<p>(Related: &quot;Stop Writing Dead Programs&quot; by Jack Rusher (Strange Loop 2022): <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8Ab3ArE8W3s&amp;t=1383s" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8Ab3ArE8W3s&amp;t=1383s</a>)