Advanced Python Features

Hey yall! Original author of the blog here!<p>I did not expect to wake up at 4am seeing my post on front page HN, but here we are nevertheless :D<p>As the intro mentioned, these started off as 14 small tweets I wrote a month prior to starting my blog. When I finally got that set up, I just thought, &quot;hey, I just spent the better part of two weeks writing these nifty Python tricks, might as well reuse them as a fun first post!&quot;<p>That&#x27;s why the flow might seem a little weird (as some pointed out, proxy properties are not really a Python &quot;feature&quot; in of itself). They were just whatever I found cool that day. I tried to find something more esoteric if it was a Friday, and something useful if it was a Monday. I was also kinda improving the entire series as it was going on, so that was also a factor.<p>Same goes with the title. These were just 14 feature I found interesting while writing Python both professionally and as a hobby. Some people mentioned these are not very &quot;advanced&quot; per se, and fair enough. I think I spent a total of 5 second thinking of a title. Oh well!

Every time I try to use Python I get this mixed feeling of liking how few guard rails there are between me and the logic I am making and this lingering worry that my code looks like fools attempt to use Python. So much exists as convention or loosely followed rules. Whenever I read articles like this I am wowed by the depth of things I didn&#x27;t know about Python or how much has changed. It makes something like Go feel like a comfort blanket because I have reasonable certainty code I write won&#x27;t feel deprecated or left behind a year or two later. Excellent article showing off even more I didn&#x27;t know.

My own opinion is that Python shall remain Python, and golang, Rust and Typescript each should be whichever they are with their unique philosophy and design.<p>I am coding in all 4, with some roughly 28 years now, and I don&#x27;t like what is becoming of Python<p>There is a reason why python become that popular and widely adapted and used, and it is not the extra layers of type checking, annotations and the likes.<p>this looks familiar to me, but from other languages.<p><pre><code> response := get_user_input() </code></pre> I am aware of the factI am in minority, and not trying to change anyone&#x27;s mind, simply what this voice to be heard from time to time.<p>All in all, a very comprehensive list of some of the recent introduced features.<p>There is an older list on SO which readers might also find useful:<p><a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;101268&#x2F;hidden-features-of-python" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;101268&#x2F;hidden-features-o...</a>

Good list, its one of these things you either never heard of, or used for years and think everyone knew that. I&#x27;ll add a few:<p>* did you know __init__.py is optional nowadays?<p>* you can do relative imports with things like &quot;from ..other import foo&quot;<p>* since 3.13 there is a @deprecated decorator that does what you think it does<p>* the new generics syntax also works on methods&#x2F;functions: &quot;def method[T](...)&quot; very cool<p>* you can type kwargs with typeddicts and unpack: &quot;def fn(*kwargs: Unpack[MyKwargs])&quot;<p>* dataclasses (and pydantic) support immutable objects with: &quot;class MyModel(BaseModel, frozen=True)&quot; or &quot;@dataclass(frozen=True)&quot;<p>* class attributes on dataclasses, etc. can be defined with &quot;MY_STATIC: ClassVar[int] = 42&quot; this also supports abstract base classes (ABC)<p>* TypeVar supports binding to enforce subtypes: &quot;TypeVar[&#x27;T&#x27;, bound=X]&quot;, and also a default since 3.13: &quot;TypeVar[&#x27;T&#x27;, bound=X, default=int]&quot;<p>* @overload is especially useful for get() methods to express that the return can&#x27;t be none if the default isn&#x27;t None<p>* instead of Union[a, b] or Optional[a] you can write &quot;a | b&quot; or &quot;a | None&quot; nowadays<p>* with match you can use assert_never() to ensure exhaustive matching in a &quot;case _:&quot; block<p>* typing has reveal_type() which lets mypy print the type it thinks something is<p>* typing&#x27;s &quot;Self&quot; allows you to more properly annotate class method return types<p>* the time package has functions for monotonic clocks and others not just time()<p>anyone know more things?

This is a nice list of &quot;things you might not know&quot; that is worth skimming to add to your toolkit.<p>If you are really interested in &quot;advanced Python&quot;, though, I would recommend the book <i>Fluent Python</i> by Ramalho. I have the first edition which is still highly relevant, including the async bits (you just have to translate the coroutines into async syntax). There is a second edition which is more up to date.<p>I would also recommend checking out the functools[0] and itertools[1] modules in the standard library. Just go and read the docs on them top to bottom.<p>It&#x27;s also worth reading the first few sections of Python Data Model[2] and then bookmarking this page.<p>[0] <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;functools.html" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;functools.html</a><p>[1] <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;itertools.html" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;itertools.html</a><p>[2] <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;reference&#x2F;datamodel.html" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;reference&#x2F;datamodel.html</a>

For me, the biggest benefit to python is that it feels like executable pseudocode. The language gets out of the way of your domain level instructions. This is probably why most non-programmers have the easiest time with python.<p>The more fancy stuff you add to it, the less attractive it becomes. Sure, most of these things have some sort of use, but I reckon most people do not get deep enough into python to understand all these little things.

The only things I’d have probably change about this list is the inclusion of some of the collections.abc containers for type annotations, TypedDict and how it can make working with strictly structured dictionaries not terrible (but if it looks like an class and quacks like a class, make it a class if you can), and Counter (only because I forget it exists every single time I should have used it).<p>Several comments disliking the walrus operator, like many of the features on this list I also hated it… until I found a good use for it. I almost exclusively write strictly typed Python these days (annotations… another feature I originally hated). The walrus operator makes code <i>so much cleaner</i> when you’re dealing with Optionals (or, a Union with None). This comes up a lot with regex patterns:<p><pre><code> if (match := pattern.search(line)) is not None: print(match.group()) </code></pre> Could you evaluate match on a separate line before the conditional? Sure. But I find this is a little clearer that the intended life of match is within the conditional, making it less tempting to reuse it elsewhere.<p>Not a Python feature specifically, but I’d also love to see code that uses regex patterns to embrace named capturing groups more often. .group(“prefix”) is a lot more readable than .group(1).

Nitpick about 9.3 Short Circuit Evaluation: both things evaluate differently if you have empty strings. The if-else clause treats empty strings as valid while the or operator will treat them equivalent with None.

I would argue that most of these features (basically everything except metaclasses) are not advanced features. These are simple, but for some reason less well known or less used features.<p>Metaclasses are however quite complex (or at least lead to complex behavior) and I mostly avoid them for this reason.<p>And &#x27;Proxy Properties&#x27; are not really a feature at all. Just a specific usage of dunder methods.

As someone coming from Javascript&#x2F;Typescript &amp; now working full time in Python, this is a lovely - mostly fantastically useful - little resource. Some choice observations:<p>1. Typing overloads: TS has typed overloads I think largely as an affordance to an unfortunate feature of Javascript. In my experience overloads are an anti-pattern or at best code smell. It&#x27;s nice that you can type them if you&#x27;re cleaning up an existing codebase that uses them but I would consider them tech debt.<p>2. Keyword-only and Positional-only Arguments: This is the opposite of the 1st feature (ability to make method signatures more strict) but man is the syntax cryptically terse. I&#x27;d love to use this everywhere but I&#x27;d be concerned about readability.<p>3. Future Annotations: Thank you for this section - forward references have been a real pain for me recently &amp; this is the first explanation that&#x27;s scratches the service of the &quot;why&quot; (rather than just focusing on case-by-case solutions), which is much more helpful. Bring on PEP 649.<p>4. Generics: Cries in legacy 3.10 codebase<p>5. Protocols: As a Typescript guy, this seems very cosy &amp; familiar. And not very Pythonic. I&#x27;m not sure how to feel.<p>14. Metaclasses:<p>&gt; <i>if you are that 1% which has a unique enough problem that only metaclasses can solve, they are a powerful tool that lets you tinker with the internals of the Python object system.</i><p>OR if you&#x27;re one of the many devs that believes their problem is unique &amp; special &amp; loves to apply magical overengineered solutions to simple common problems, then the next person who inherits your code is going to really love your metaclasses. They sure make tracing codepaths fun.

This is wild and something I didn&#x27;t know about:<p><a href="https:&#x2F;&#x2F;blog.edward-li.com&#x2F;tech&#x2F;advanced-python-features&#x2F;#2-keyword-only-and-positional-only-arguments" rel="nofollow">https:&#x2F;&#x2F;blog.edward-li.com&#x2F;tech&#x2F;advanced-python-features&#x2F;#2-...</a><p><pre><code> def bar(a, &#x2F;, b): ... # == ALLOWED == bar(1, 2) # All positional bar(1, b=2) # Half positional, half keyword # == NOT ALLOWED == bar(a=1, b=2) # Cannot use keyword for positional-only parameter</code></pre>

TFA&#x27;s use-case for for&#x2F;else does not convince me:<p><pre><code> for server in servers: if server.check_availability(): primary_server = server break else: primary_server = backup_server deploy_application(primary_server) </code></pre> As it is shorter to do this:<p><pre><code> primary_server = backup_server for server in servers: if server.check_availability(): primary_server = server break deploy_application(primary_server)</code></pre>

I enjoyed reading the article. I&#x27;m far from a Python expert, but as an observation, most of these features are actually just typing module features. In particular, I wasn&#x27;t sold on Generics or Protocols as I would have just used duck typing in both cases... Does modern, production-level python code use types everywhere? Is duck typing frowned upon?

Working in the ML field, I can&#x27;t hate Python. But the type system (pre-3.12, of course) cost me <i>a lot</i> of nerves. Hoping for a better post-3.12 experience once all libraries are usable in 3.12+. After that experience, I’ve come to truly appreciate TypeScript’s type system. Never thought I’d say that.

Turns out I managed to use almost all of these during a refactor of a project at work, even metaclasses... (Metaclass usage is justified in my case: we have a sort of language evaluator and using a metaclass lets us define function arguments with their types and validators in a very coincise and obvious way similar to Pydantic.)<p>I think this list should also include descriptors[0]: it&#x27;s another metaprogramming feature that allows running code when accessing or setting class attributes similar to @property but more powerful. (edit: nvm, I saw that they are covered in the proxy properties section!)<p>I think the type system is quite good actually, even if you end up having to sidestep it when doing this kind of meta-programming. The errors I do get are generally the library&#x27;s fault (old versions of SQLAlchemy make it impossible to assign types anywhere...) and there&#x27;s a few gotchas (like mutable collections being invariant, so if you take a list as an argument you may have to type it as `Sequence[]` or you&#x27;ll get type errors) but it&#x27;s functional and makes the language usable for me.<p>I stopped using Ruby because upstream would not commit on type checking (yes I know you have a few choices if you want typing, but they&#x27;re a bit too much overhead for what I usually use Ruby for, which is writing scripts), and I&#x27;m glad Python is committing here.<p>[0]: <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;howto&#x2F;descriptor.html" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;howto&#x2F;descriptor.html</a>

Here is another one, list and expression comprehensions shared with ML languages (not that AI one), apparently many aren&#x27;t aware of them.<p>The itertools package.

Python has evolved into quite a complex language considering the amount of features that come built-in. The documentation, while complete, does not facilitate discovery of many of those features.

&gt; @overload is a decorator from Python’s typing module that lets you define multiple signatures for the same function.<p>I&#x27;ll be honest, I&#x27;ve never understood this language feature (it exists in several languages). Can someone honestly help me understand? When is a function with many potential signatures more clear than just having separate function names?

Nice!<p>I would add assert_never to the pattern matching section for exhaustiveness checks: <a href="https:&#x2F;&#x2F;typing.python.org&#x2F;en&#x2F;latest&#x2F;guides&#x2F;unreachable.html#assert-never-and-exhaustiveness-checking" rel="nofollow">https:&#x2F;&#x2F;typing.python.org&#x2F;en&#x2F;latest&#x2F;guides&#x2F;unreachable.html#...</a>

The way the language is evolving, it seems likely to me that people in the applications camp (ML, simple web-dev, etc.) will soon need a &quot;simple Python&quot; fork or at least an agreed-upon subset of the language that doesn&#x27;t have most of these complications (f-strings are a major success, though).

Some of these newer features don&#x27;t seem like improvements to the language (e.g. the Walrus operator):<p>&#x27;&#x27;&#x27;<p># ===== Don&#x27;t write this =====<p>response = get_user_input()<p>if response:<p><pre><code> print(&#x27;You pressed:&#x27;, response) </code></pre> else:<p><pre><code> print(&#x27;You pressed nothing&#x27;) </code></pre> # ===== Write this instead =====<p>if response := get_user_input():<p><pre><code> print(&#x27;You pressed:&#x27;, response) </code></pre> else:<p><pre><code> print(&#x27;You pressed nothing&#x27;)</code></pre> &#x27;&#x27;&#x27;<p>The first implementation is immediately clear, even if you&#x27;re not familiar with Python syntax. If you don&#x27;t know what the &quot;:=&quot; operator does, the code becomes less readable, and code clarity is traded away in favor of being slightly more concise.

I thought the typing overload was interesting because I thought it was going to be like overloading in C# and Java but it&#x27;s really just useful for documentation and intellisense tools in IDEs.

Good article! I found the typing bits particularly interesting. This part of the language has been evolving rapidly recently, which is cool if you like static typing.<p>Although I think the example of type alises in section 4 is not quite right. NewType creates a new &quot;subtype&quot; which is not equivalent to the original type. That&#x27;s different to TypeAlias, which simply assigns a name to an existing type. Hence NewType is still useful in Python 3.12+.

Good list. Some of these i knew already, but the typing overloading and keyword&#x2F;positional-only arguments were new to me.<p>One personal favorite of mine is __all__ for use in __init__.py files. It specifies which items are imported whenever uses from x import *. Especially useful when you have other people working on your codebase with the tendency to always import everything, which is rarely a good idea.

not sure whether to laugh or cry at this ... looks like over the years python has agglomerated some of the features raku has had from day 1 (2015) in sadly inconsistent ways, often via module includes:<p><pre><code> - multi subs &amp; methods = &quot;typing overloads&quot; - named &amp; positional args - stubs = &quot;future annotations&quot; - subsets = &quot;Generics&quot; - protocols - wrappers = &quot;context managers&quot; - given &#x2F; when = &quot;structural pattern matching&quot; - my declaration = &quot;walrus op&quot; - short circuit evaluation - operator chaining - fmt - concurrency </code></pre> If you are a Python coder and you feel the need for some of this, I humbly suggest you take a look at <a href="https:&#x2F;&#x2F;raku.org" rel="nofollow">https:&#x2F;&#x2F;raku.org</a>

My favorite hack is to use a single threaded ThreadPoolExecutor as my main thread. When another thread wants to communicate with it, it just queues the callback function into main using submit. This cleans up the architecture and avoids the use of Queue which is a bit slower.

It is a great article, but it completely misses what&#x27;s new in Python parallelism with the GIL being optional since 3.13, also multiple interpreters via InterpreterPoolExecutor coming in 3.14.

This sounds fun if you have 10x programmers or at least IQ &gt; 140 programmers in charge. Last place I worked, I was told never use &quot;smart&quot; tricks if you can do the same thing in a simpler way. For-else and f-strings and := sound like harmless enough (though the last one is controversial); &quot;with&quot; is useful for resources that need to be deallocated but &quot;yield&quot;? Really? &quot;more readable&quot; when you slap a decorator on the thing? Why are we packing the enter and exit operations into a single function?<p>Playing with the typesystem and generics like this makes me worry I&#x27;m about to have a panic attack.<p>Give me code that I can understand and debug easily, even when I didn&#x27;t write it; don&#x27;t do implicit magical control flow changes unless you have a very good excuse, and then document both the how and the why - and you&#x27;ll get a product that launches earlier and has fewer bugs.<p>Sometimes, a few more if statements here and there make code that is easier to understand, even if there&#x27;s a clever hack that could cut a line or two here and there.

This is all fun and games unless you have to debug someone elses code and they use a new feature that you didnt know about. Speaking for myself, I would be glad if there were a python_light version of the interpreter that has a simple syntax only like the early 3.x versions.<p>just my 2 ct

If context managers are considered advanced I despair at the code you&#x27;re writing.

Trying so hard to make it a typed language

Another blog with no feed :(

Those are basic, you can decompile a function and change its bytecode, you can completely revamp the parser using codecs, etc