Ask HN: How did Python become the lingua franca of ML/AI?

NumPy, SciPy and the ecosystem around them. So much of what you do in ML involves matrix operations. People used to do this stuff in Matlab. Matlab is good at numerics but it&#x27;s not a very good programming language, and doesn&#x27;t have very good libraries outside of the numeric domain. The open source nature of NumPy and Python encouraged a big open source community that is hard to get going if you&#x27;re building open source on top of a language that costs thousands of dollars per seat.<p>Python&#x27;s dynamic nature also made a lot of what&#x27;s in NumPy and the various ML libraries possible or more convenient to use. The performance is not as much of an issue if you start thinking in NumPy terms, doing operations on whole arrays where the loops are then in C. Really, Python itself is just acting as orchestration for a bunch of C code that&#x27;s doing all the work. In the case of something like Tensorflow or PyTorch, it&#x27;s actually a bunch of CUDA code that&#x27;s doing all the work and orchestrated by Python.

My last job was at an ML company.<p>Most ML people there cannot build large robust systems and some struggled with the non-algorithmic bits of software. I am sure that some can out there in the world, but for the most part our ML people were very good at creating models and not very good at the development part, especially as the program grew (part of the motivation to hire devs like me in the first place).<p>Python gets rid of as much of the developmental complexity as possible. No types, no memory management, libraries for everything, No need to create a class to run &quot;hello world.&quot; Pip makes it trivial to import things. Use PyCharm and you just need to click the run button, with no complicated JRE and JDK setup.<p>It is the fastest way to start writing models.

Python&#x27;s a lingua franca in AI&#x2F;NN because it was already a dominant language in scientific computing. Its dominance in scientific computing grew steadily through the 1990s and 2000s, for a few reasons:<p>1) Python -- specifically CPython -- made it easy to wrap existing, thoroughly tested high performance libraries in Python APIs. So, you got easy access to things like GSL and BLAS and LAPACK, but you get to call numpy.linalg.svd instead of GESDD.<p>2) Python was a general purpose language, unlike R or MATLAB, so you could extend existing systems to do more without running into a wall.<p>3) Python was a heck of a lot less effort to use than C++.

I don&#x27;t think there&#x27;s a master plan or a design idea about that.<p>There&#x27;s an old saying that goes like &quot;Python is the second best language for anything&quot;.<p>Python isn&#x27;t the best for any kind of task; but you can do almost anything in any field with python and some libraries. It&#x27;s reasonably easy for a non-programmer to use it.<p>I think my first experience with GPU programming was using CUDA with C (I think it was kind of customized C in mid-2000s), so Python is not there since forever.<p>But if you need to do a bit of web scraping&#x2F;input data manipulation, a bit of &quot;offering a gui&quot; (e.g. a small web server that shows the data), a bit a of matrix&#x2F;vectorized operations, a bit of model training or even just inference... python has everything and everything is reasonably good. At least some of those operations would be cumbersome in other programming languages.<p>Try using R for general-purpose programming. Or Java for number crunching&#x2F;matrix operations. They just suck.<p>Try finding the &quot;greatest common divisor&quot;, functionality-wise, for the many tasks that you need in a ML system (just as many other systems), and you&#x27;ll find Python.<p>The drawback is, IMHO, that it doesn&#x27;t &quot;scale&quot; well. Python makes great proof of concepts and prototypes, but I&#x27;ll always pick a different stack (possibly with multiple languages and technologies) if I want a long-running, maintainable production system.

The same reason that Python is heavily used in scientific computing.<p>ML&#x2F;AI&#x2F;Scientists aren&#x27;t systems people. They don&#x27;t want to care about memory management&#x2F;parallelization&#x2F;etc. - they want to write perfect little mathematical poems which get executed on a perfect Turing machine.<p>Python is good at that. Thanks to the efforts of actual systems people, its libraries (numpy, scipy, etc.) run quick enough to be practical on a lot of workloads.

Another way to analyze the problem: what other language would it have been, given the moment ml hit?<p>You say compared to other scripting languages&#x27;. Let&#x27;s list them.<p>Ruby: no numeric support Go: unnecessary typing, modest numeric support, shitty generics Bash: ha ha ha Scala, java, c, cpp: not a scripting language, complex Tcl, php: out of favor Rust: hadn&#x27;t happened yet R: in memory bias, not as simple Other languages were obscure or owned by monoliths (kotlin, swift, c#)<p>Python also has multiple implementations, a minor thing, but not really. Pypy keeps cython on its toes.<p>C# really could be a contender. I am more productive in c# than any other language except python (although I think I will be more productive in rust)<p>Python is, almost unarguably, the easiest language to code in, right now, period. It has the greatest expressiveness and the simplest syntax. I use it for large scale open source art projects, and you can use it for ai.<p>Why are you asking?

I can only provide anecdotal material, but back in 2007-2014 when I was a particle physics researcher, we saw a high uptake of python for steering data analysis jobs. The actual calculations were done in C++. Gradually over the years, as more students joined the LHC conquest, our tools evolved to allow more of the analyses to be directly programmed in Python. R was never a thing among the 10000+ physicists in our community. These people have since then drifted around the world working on Big Data, ML and recently Data Science. It’s hard to keep count, but I routinely recognize fellow particle physicists at various ML companies.<p>For the curious, our primary hammer was “ROOT” <a href="https:&#x2F;&#x2F;root.cern" rel="nofollow">https:&#x2F;&#x2F;root.cern</a> - note its well-evolved ability to connect Python and C++ code.

I have a historical perspective on this topic. Data Science popularity was rapidly growing and &quot;R&quot; was the lingua franca around 2010 - 2014.<p>I attended Strata conference in 2014 and after visiting various technology exhibition booths there, I saw a common theme: tech companies were building data solutions using Python as R was no good for the purpose<p>In a meeting scheduled to share my take-aways from the conference, I predicted &quot;Python will emerge to be the language of DataScience in few years&quot;

Numpy, Scipy pandas, etc are a way to use scripty syntax to write CPP code.<p>Under the hood you get the benefits of CPP: stuff is dense in cache, operations are efficient.<p>But you can write it without a bunch of types, templates and allocators, which confuse people who aren&#x27;t used to it. Most numeric code doesn&#x27;t have a load of types anyway, it&#x27;s just a few operations on some very large matrices.<p>Add to that the benefit that you can just ask of python&#x27;s universe of libraries, which is quite large compared to rivals like MATLAB or R. Want to serve your model as a website? Jam it into Flask. Need crypto lib to grab the data? No problem, just pip it and import.

Python has always been used as a nice layer over various C libraries so when ML started taking off and people started using GPUs to accelerate training and inference it was a natural choice for acting as the high level code for interfacing with the low level GPU code.<p>There were some other DSLs that were being developed at the time but the ones that stuck were the Python ones. [1]<p>1: <a href="https:&#x2F;&#x2F;terralang.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;terralang.org&#x2F;</a>

Pythons&#x27; USP was and still is its ability to provide a simple &amp; intuitive &quot;glue layer&quot; over lower-level libraries. Most of the performance-critical functionality that Python relies on for ML is written in C&#x2F;C++&#x2F;Fortran and Python mostly provides the UI layer (this is an oversimplification of course).<p>Wrapper generators and compiler tools like Cython and before that SWIG made it very easy to glue existing functionality to Python, so together with Pythons&#x27; great usability and user-friendly language it created a killer combination for productive data science &amp; ML.<p>That said other languages could&#x27;ve pulled this off as well, Ruby for example. Python had more early traction in the scientific and high-performance computing communities though whereas Ruby was more popular in web development (due to Rails), which ultimately gave Python the edge and attracted more and more toolmakers to its ecosystem, which in turn spurred further growth. Great &quot;IDEs&quot; like the iPython&#x2F;Juypter notebook were also a key factor in Pythons&#x27; success, as they provided a super user-friendly UI for data scientists.

Because Python has NumPy, which implements vectorized math on arrays and matrices. Machine learning algorithms are implemented naturally and efficiently with those primitives. PyTorch, TensorFlow, and I think every other machine learning framework in Python all use NumPy.<p>JavaScript, Ruby, and Perl either don&#x27;t have this abstraction at all, or they have much weaker versions of it, and many fewer scientific libraries.<p>NumPy started in the early 2000&#x27;s and continues to this day. It takes decades to build up this infrastructure! This recent interview with NumPy creator Travis Oliphant is great:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=gFEE3w7F0ww" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=gFEE3w7F0ww</a><p>He talks about how there were competing abstractions like &quot;Numeric&quot; and another library, and his goal with NumPy was to unify them. And how there are still some open design issues &#x2F; regrets.<p>There were multiple people in the nascent Python community who were tired of MATLAB, not just because it&#x27;s proprietary, but because it&#x27;s a weak and inefficient language for anything other than its scientific use cases. You won&#x27;t have a good time trying to write a web app wrapper in MATLAB, for example.<p>The much more recent Julia language is also inspired positively and negatively by MATLAB, and is very suitable for machine learning, though it doesn&#x27;t have the decades of libraries that Python has.<p>-----<p>The NumPy extension was in turn enabled by operator overloading in Python (which is actually a very C++ influenced mechanism). JavaScript doesn&#x27;t have operator overloading; I&#x27;m pretty sure Perl doesn&#x27;t, but not sure about Ruby. Lua and Tcl do not have it. (Lua does have a machine learning framework though -- <a href="http:&#x2F;&#x2F;torch.ch&#x2F;" rel="nofollow">http:&#x2F;&#x2F;torch.ch&#x2F;</a> -- but I think PyTorch is more popular now.)<p>So if Guido didn&#x27;t design Python with operator overloading, then NumPy would not have grown out of it.<p>Also relevant is Guy Steele&#x27;s famous talk <i>Growing a Language</i> (late 90&#x27;s or early 2000&#x27;s I think). He advocates for operator overloading in Java so end users can evolve language with their domain expertise! Well Java never got it, and Python ended up having the capabilities to grow linear algebra.<p>Guido has even said he doesn&#x27;t really use or even &quot;get&quot; NumPy! So it turns out that an extensible design does have the benefits that Steele suggested (although it&#x27;s a very difficult language design problem.) There have been several enhancements to Python driven by the NumPy community, like slicing syntax and semantics and the @ matrix multiplication operator. And I think many parts of the C API like buffers.<p>-----<p>Another interesting thing from Oliphant&#x27;s interview is that he really liked that Python has complex numbers. (I don&#x27;t think any of JavaScript, Ruby, Perl, or Lua have them in the core, which is important.) That piqued his interest and kicked off a few decades of hacking on Python.<p>He was an electrical engineering Ph.D. student and professor, and complex numbers are ubiquitous in that domain. Example:<p><pre><code> $ python3 -c &#x27;print(3j * 2 + 1)&#x27; (1+6j) </code></pre> This is another simple type built on Python&#x27;s extensible core, and it&#x27;s short.<p><pre><code> $ Python-3.9.4$ wc -l Objects&#x2F;complexobject.c 1125 Objects&#x2F;complexobject.c </code></pre> I recommend writing a Python extension in C if you want to see how it works. See Modules&#x2F;xx*.c in the Python source code for some templates &#x2F; examples. IMO the Python source code is a lot more approachable than Perl, Ruby, or any JS engine I&#x27;ve looked at.

At the time when ML craze hit, python was already very popular as a beginners language and had good numeric libraries.<p>Many people of STEM fields without any programming background, had their first taste of programming with python. And it caught on.<p>Also, the real stuff is probably written in C&#x2F;C++&#x2F;CUDA&#x2F;ASM. Its only the interface that is python (because of its inertial popularity)

The use of python in most ML&#x2F;AI research is not even a &quot;glue&quot; language.<p>It is used as a <i>shell</i>. It&#x27;s merely an interface to some gigantic, highly optimized libraries (numpy, scipy, and later, Tensorflow, Pytorch, etc.), and it does a very decent job at being an interface.<p>- The language is easy to grasp, at least the part that is used in data science and ML;<p>- The syntax is &quot;familiar&quot;, as compared with R;<p>- There are many more general purpose libraries in Python than in R;<p>- There&#x27;s no memory management problems;<p>- The standard library is packed with batteries;<p>- No compiling, which is important for being a shell;<p>- It&#x27;s better than bash etc. at dealing with non-text data, especially numerical values;<p>- The community was already writing extensions in C;<p>Some other language <i>could</i> work well, too, had someone written a numpy for it at the time. But there really aren&#x27;t that many people who are capable, interested, and invested enough to write such a marvelous library.

This happened when MIT switched from Scheme to Python some time in the &#x27;00s. Python&#x27;s adaptation increased in the scientific community further and here we are.

Python had basically already won numerical computing thanks to NumPy, SciPy, and Matplotlib before ML really blew up. The other two serious contenders were R and Matlab. Python is a much better general purpose language than either of those, and Matlab is proprietary.

1. It&#x27;s extremely easy. Before the so-called revolution and CS people trying to get into it in droves, it was a niche topic dominated by the lifelong-researcher types. They could not be bothered with complex code. Writing code should not get in the way. Then, Lisp dominated the ML&#x2F;AI scene. Now Python does, for this reason, to some extent. Python being easy is also helpful for non-CS engineering and other science grads to learn quickly.<p>2. Python has a huge ecosystem. NumPy, SciPy, and now Tensorflow, PyTorch, JAX. These makes lives easier.<p>3. Python and its ecosystem is FOSS. Students, hobbyists can learn it for free. (Quick anecdote: my uni in India, a very reputed non-IIT one, with sub-optimal funding, two years ago switched to Python + ecosystem for Physics and CS courses- both major and minor. This switch happened directly from C. Before that, Fortran was used. MATLAB, SPSS, etc. was never an option for cash-starved Indian unis. This is pretty much the same all across India. And thus you get a huge talent pool already trained in Python that pass-outs from hard-to-get-into unis.)<p>4. Python being general purpose also helps vis-a-vis R. R is heavily constrained. You cannot do much in it. R is used in anaalysis and Data Science. I have never seen it being used in ML, DL or RL. You learn Python, you can do non-trivial file manipulation in it. Good luck doing that with R or MATLAB.<p>5. The amount of people who needs to write code that reaches the metal is very small. I never needed to look under the sheets. I spend my life writing PyTorch, fastai, and TFLite. A friend of mine doing PhD needed to write custom CUDA code and then a wrapper so that it could be accessed from Python. He said that it was a very horrible experience. But the number of such people too little to bring Julia to mainstream. Julia removes the &quot;two-language problem&quot;, but most people never need to use anything besides Python.

It comes down to timing, really. Just like most technology fads.<p>Python is interpreted&#x2F;dynamic, open source, general-purpose, relatively popular, and is easy to write low-overhead wrappers to C&#x2F;C++&#x2F;Fortran libraries. In 2008-2010, when ML took off, Python was the only language with these properties.<p>Python, however, has its problems: an atrocious concurrency story (GIL, colored coroutines, asyncio vs trio rift, fork is inefficient thanks to GC, etc), highly non-compostable (especially in scientific computing), and package management is broken. The language is also inherently slow (unfixable). It will be replaced by something else eventually.

I would put it mostly down to Spark.<p>Originally, it was only available in Scala&#x2F;Java but then they added Python support courtesy of Py4J. And since Python was massively simpler than Scala it exploded in popularity very quickly becoming the default language.<p>So then you had Data Scientists who were already writing a lot of data transformations in Spark looking around at the rest of the Python ecosystem finding libraries like pandas, IDEs like Jupyter and basically staying there since it was so much easier than alternatives.<p>Their interests aren&#x27;t really in computer science and so they look for whatever language can get them to an outcome as quickly and easily as possible. Even if it&#x27;s not the most optimal, elegant or maintainable.

Because a lot of the early development in these areas was done by mathematicians and physicists who weren’t programmers (and who had less exposure to languages). These are folks who just wanted an answer to a question or a premise, and the elegance of the path that took them to the answer was utterly insignificant.<p>In some cases you might see a 3000 line python script with no defined functions… just loops and conditionals and lots of copy-pasted code with small variations in each section.<p>It’s really a shame, since there are so many more elegant languages which are equally or more powerful. But python is not a terrible language… it’s just an everyman get-shit-done language. We could be worse off.

I suspect some of the popularity came from IT and engineering departments preferring to run Python compared to, say, Matlab, Excel, or some other GUI based application not designed to run on servers. R also has a lot of adoption but I think Python is a little more natural to run since so many infrastructure tools are written in Python<p>It also works fairly well cross platform. That means you can develop on Windows and run on Linux without too many issues (at least for ML&#x2F;AI stuff, common frameworks usually have per platform binaries published)<p>i.e. Python is familiar to the people supporting production systems

&gt;I bet there&#x27;s some interesting history here.<p>ML &#x2F; AI were derived from Data Science. So hence it was built up upon the same python foundation.<p>As to why Python on Data Science. One needs to be reminded most people doing Data Science, or any Matlab type of work do not considered themselves as programmers. They dont want to learn about 20 reason why functional programming, or objected oriented programming are better and 100s other best practice with 1000 tricks to write the same program.<p>Although I do wonder if Julia may have a chance to dethrone it in the next 10 years.

I saw Fortran being wrapped in it because it could be compiled down into libraries with C-compatible ABIs, and Python can load C libraries directly without fuss.<p>A lot of NumPy and SciPy ensued, and the rest is history.

My narrow perspective is that Python is the one language that both companies engaged in ML research (like Google) have been using, and also a very common language of instruction, for instance in the CS program at the school where I studied. If you were a CS student interested in ML&#x2F;AI, starting school ten to fifteen years ago, but not particularly interested in software engineering, you&#x27;d be able to get by with only really knowing Python. Depending on how widespread this is, I&#x27;m guessing it&#x27;s a part of the picture.

Python became the <i>de facto</i> glue language for supercomputing a very long time ago because you could easily bind C code into it. If you needed linear algebra etc to run on a massive supercomputer, there were highly optimized Python libraries for that so the researcher didn’t have to write C&#x2F;C++&#x2F;Fortran. This massively improved iteration times for a lot of scientific computing efforts with only a modest loss of performance. By the time data science&#x2F;ML&#x2F;AI&#x2F;etc became a thing these tools were already very mature and also relevant.<p>The tl;dr: Python had the advantage of a mature legacy in supercomputing doing many of the same types of computations done in AI&#x2F;ML. Those libraries and bindings provided a massive leg up versus other scripting languages that did not have this kind of capability effectively built-in.

Python is the default in many fields beside ML, however ML has its own very efficient languages, I recently discovered octave through a course I attended. Really worth having a look at <a href="https:&#x2F;&#x2F;www.gnu.org&#x2F;software&#x2F;octave&#x2F;index" rel="nofollow">https:&#x2F;&#x2F;www.gnu.org&#x2F;software&#x2F;octave&#x2F;index</a> is mostly compatible with matlab.

I think there was a strong cohort of scientists using matlab, and python and numpy adopt the same language conventions. Going from matlab to python is effortless.<p>Moreover, scientists are typically so-so programmers so not having to worry about complexities like dereferencing pointers, specifying types etc, makes the language much easier to pick up.

At least in academic circles python has always looked like pseudo code for C(or similar) you can execute so everyone has &quot;used it&quot; at some point or another to describe algorithms and stuff. Then the same academic circles do a lot of ML&#x2F;AI research so python had a natural advantage.

Python was already the lingua franca before the whole deep learning&#x2F;AI thing. It has numpy&#x2F;scipy&#x2F;pandas&#x2F;scikit-learn, etc. And when did numpy happen? It was in 1996.<p>Arguably its biggest competitor then was R, but R is not well accepted by programmers. Yet another alternative is Matlab, but OMG, using matlab for anything string related is killing me.<p>While there is some history to it, Python won in the end isn&#x27;t a surprise to anyone. It is simple but not toyish for real world system. I am working in one of the big techs, and Python is running the production workload for most AI services just fine.<p>I took a LOT of issue with dynamic typing, but for ML&#x2F;AL you are going to write a lot ad-hoc data wrangling code, sometimes even Python feels verbose.<p>TL;DR: It had already won.