Instagram Makes a Smooth Move to Python 3

“Yeah, Python is great in so many ways, too bad it’s not really scalable.”<p>I&#x27;m not even sure what this means anymore. I guess I&#x27;m just not sure how any language, when used correctly, could be inherently unscalable. My guess is statements like this came from a time when monoliths were the application design of choice? Now, assuming Instagram has just 1,000 photo handling servers, each one is only responsible for 95,000 photos a day.<p>Of course, that&#x27;s not to say that Instagram doesn&#x27;t have CAP issues. It does, especially in the &quot;C&quot; area, but again, not a problem inherent in the language.

Members of the Py3 transition team (the authors of that article) gave a talk about the project at PyCon 2017: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=66XoCk79kjM" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=66XoCk79kjM</a>

I can&#x27;t say I think this has ever been really true: &quot;Performance speed is no longer the primary worry. Time to market speed is.&quot;<p>Performance has been a concern but programatic loadbalancing has been around for decades. When I worked at MSN&#x2F;Linkexchange back in the late 90&#x27;s we never really worried heavily about the performance of the language we used (Perl) because we could scale out servers. Perl isn&#x27;t that speedy but it sure was easy to develop in. We served a billion and a half clicks per month with 8-10 machines from a single datacenter before I left, with Perl.

Right, you hear a lot of griping about moving from Python 2 to 3 but I personally didn&#x27;t have as much trouble as expected. Some of my projects just worked. One small tip I don&#x27;t think they mentioned. Start using the logging module instead of print and it will eliminate one class of potential issues.

I&#x27;ve been writing mostly small python projects for 15 years, starting with 2.2.<p>I&#x27;ve had no issues in the migration to 3.<p>I did some building with it around 3.3, starting to commit around 3.4, and with 3.5 I build everything in it.<p>I don&#x27;t have the performance challenges Instagram has, but my experience with application development in general is that 98% of performance challenges can be solved with (not-too) clever engineering. This applies to projects in every language.<p>There are a vanishingly small number of scenarios where the performance of your runtime actually dictates your performance limits.<p>If you&#x27;re working on something and are worried about Python&#x27;s performance, or which Python to use, don&#x27;t. Use 3, optimize later.

I watched the PyCon keynote on this topic, and while it&#x27;s nice to hear they&#x27;ve moved to Python3 their approach probably shouldn&#x27;t be copied.<p>For example, in their codebase they had ambiguity between bytestrings and unicode strings. As Python3 tries to prevent you doing this, to resolve a big footgun from Python2.<p>The right fix here is to be consistent in your use of strings. Sometimes that is tricky because of how third party libraries have decided to implement their 2&#x2F;3 compatibility, but it helps prevent shooting yourself in the foot with unicode bugs down the line.<p>Instagram did not do this. They created utility functions to force their data into the format they wanted at the point it is used. In other places they used tuple() to make sure that map calls that had side effects were fully iterated over.<p>In short, they had bad Python2 code and now have had Python3 code. Sometimes, at large scale, it&#x27;s your only choice. But to smaller companies looking at this it&#x27;s a bad idea. You&#x27;re setting a precedent in your code that it&#x27;s okay to make the same mistakes that Python3 tried to prevent.

So their server needs are growing faster than their user-base to the extent that they considered switching languages. PHP didn&#x27;t seem to perform much better, so they stuck with python and got a ~12% CPU usage improvement by moving to python3. It doesn&#x27;t seem like a 12% one-time improvement actually solves the original problem, though. Perhaps pypy would have been better?

types in python 3.5?! I had no idea -- that&#x27;s exciting.

&gt;<i>It made sense that, if we were going to stay on Python for the next ten years, we should invest in the latest version of the language.</i><p>I don&#x27;t know this stands out to me in particular, but the 10-year commitment is definitely a big decision. I suppose I&#x27;ve never had to make a similar decision so perhaps this is more common than I think.