Pandas 2.0

I&#x27;m curious if there will be any appreciable performance gains here that are worthwhile. FWIW, last I checked[0], Polars still smokes Pandas in basically every way.<p>[0] <a href="https:&#x2F;&#x2F;www.pola.rs&#x2F;benchmarks.html" rel="nofollow">https:&#x2F;&#x2F;www.pola.rs&#x2F;benchmarks.html</a>

Jeff Reback gave a presentation on the roadmap for Pandas at PyData NYC 2022 [0]. In it, he basically says that pandas is used so widely in industry that big breaking changes are a non-starter, there won&#x27;t be any radical changes to the API, but more performant implementations can&#x2F;will be built into the library (although not set as defaults, at least not for a long while). Not a revolutionary leap, but a move towards making Wes McKinney&#x27;s Arrow work more accessible through pandas.<p>[0] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=85XdsWz_Q_o">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=85XdsWz_Q_o</a>

I know arrow support is only part way their with this release - but this is a huge deal for Pandas, for standardisation as whole, but also speed ups.<p>Benchmarking that was shared a while back here suggests 2x speed ups in some cases, 30x if you count strings since pandas uses python&#x27;s in-built string data type[1]<p>[1]<a href="https:&#x2F;&#x2F;datapythonista.me&#x2F;blog&#x2F;pandas-20-and-the-arrow-revolution-part-i" rel="nofollow">https:&#x2F;&#x2F;datapythonista.me&#x2F;blog&#x2F;pandas-20-and-the-arrow-revol...</a>

What&#x27;s new here [0], saved you a click.<p>[0]: <a href="https:&#x2F;&#x2F;pandas.pydata.org&#x2F;pandas-docs&#x2F;version&#x2F;2.0&#x2F;whatsnew&#x2F;v2.0.0.html" rel="nofollow">https:&#x2F;&#x2F;pandas.pydata.org&#x2F;pandas-docs&#x2F;version&#x2F;2.0&#x2F;whatsnew&#x2F;v...</a>

A quick skim shows a lot of quality of life improvements. Unless I am misreading, it looks like it is still a numpy backed by default. I thought one of the drivers for the 2.0 was to make Arrow the default.

strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to.<p>I&#x27;m excited to try out the new pyarrow dtypes, but it also sounds confusing that there are now 2 classes of types

Say what you will about sql, polars, pyspark, or whatever else. But nothing beats pandas&#x27; df[col].value_counts().value_counts()

How are people managing the existence of data frame APIs like pandas&#x2F;polars with SQL engines like BigQuery, Snowflake, and DuckDB?<p>Most of my notebooks are a mix of SQL and Python: SQL for most processing, dump the results as a pandas dataframe (via <a href="https:&#x2F;&#x2F;github.com&#x2F;ploomber&#x2F;jupysql">https:&#x2F;&#x2F;github.com&#x2F;ploomber&#x2F;jupysql</a>) and then use Python for operations that are difficult to express with SQL (or that I don&#x27;t know how to do it), so I end up with 80% SQL, 20% Python.<p>Unsure if this is the best workflow but it&#x27;s the most efficient one I&#x27;ve come up with.<p>Disclaimer: my team develops JupySQL.

&gt; Accessing a single column of a DataFrame as a Series (e.g. df[&quot;col&quot;]) now always returns a new object every time it is constructed when Copy-on-Write is enabled (not returning multiple times an identical, cached Series object). This ensures that those Series objects correctly follow the Copy-on-Write rules (GH49450)<p>Is this going to mean I can’t do df[‘a’] = 2 to set all values in column a to 2?

I&#x27;d love to know when geopandas snaps into alignment

How I wished they just changed the .apply for adding progress and parallelization by default instead of resorting to tqdm &amp; swifter&#x2F;dask or what have you

Is it correct that I associate to a type of Chinese bears when I read about this project? Or is Pandas an acronym?

If interested in benchmarks comparing different dataframe implementations, here is one:<p><a href="https:&#x2F;&#x2F;h2oai.github.io&#x2F;db-benchmark&#x2F;" rel="nofollow">https:&#x2F;&#x2F;h2oai.github.io&#x2F;db-benchmark&#x2F;</a>

You want to link to the actual release notes: <a href="https:&#x2F;&#x2F;pandas.pydata.org&#x2F;pandas-docs&#x2F;version&#x2F;2.0&#x2F;whatsnew&#x2F;v2.0.0.html" rel="nofollow">https:&#x2F;&#x2F;pandas.pydata.org&#x2F;pandas-docs&#x2F;version&#x2F;2.0&#x2F;whatsnew&#x2F;v...</a>

Not a lot of people realize that Pandas was inspired by R, and in particular the Tidyverse model of handling rectangular data frames, created originally by Hadley Wickham. These days R is primarily used by data scientists in academia and certain niche industries like pharma, but its impact goes way beyond its core user base.

I still prefer Stata if I am completely honest with myself.