Currently I am working on a Python code base with 10k - 100k loc. The service does not depend on Python SciPy tooling or similar but is web backend on top a large db. I worked with Python for several years and it is still my go to language for a lot of task. However I feel that Python advantages like dynamic typing become increasingly disadvantages when the code base grows. How do you deal with Python in large code bases? We use MyPy, black, Tox, pytest etc. any tips? I feel like (for the next project) moving to a typed language like Kotlin/Go etc. might be the better choice then using Python when not depending on the SciPy/ML/DL stack.
Python _is_ typed. Python is Dynamically Typed. The word "Dynamically" does not suddenly erase the word "Typed". Even CPython is Dynamically Typed.<p>You presumably mean Statically, Manifestly Typed.<p>Assembly Language and Forth are Untyped. The closest thing they have to types are bytes and cells.<p>For examples of a language that is Statically but not Manifestly Typed, have a look at Shedskin (which is a dialect of Python) or Haskell (which is a fascinating Functional, Lazy language with a regrettably poor compiler).<p>I continue to see mypy, or something like it, as the best way of taking a medium sized Python project into the world of large projects. For smaller projects, ruff or pylint are sufficient, but for large projects, mypy and similar are the way to go.
If databases are in play, instead of trying to understand the business logic, have a go at understanding the database structure first.<p>What are your tables, how they are related etc.<p>Then look at the modules that interact with the data layer directly and move up.<p>This is a language agnostic approach though, but it has worked out well for me.
What exactly are your problem's? It is hard to give advice without knowing. To be honest I don't think it makes things easier if you switch to Go, since less verbose.
How are you finding mypy for your project?<p>I find that I waste a lot of time telling mypy to ignore some library that it doesn't understand, much more than I'm saving by catching type errors (which is never). People add it to projects because it seems like a best practice and the proper thing to do, but I'm kind of unconviced that it's making things better.
The larger your codebase gets the more bazel becomes a requirement. Bazel is really not negotiable for large python code bases. The more bazel is put off, the more pain you will endure before you eventually are forced to use bazel. You <i>will</i> be forced to use bazel or a system like it because eventually your good devs will not tolerate your codebase and leave without it.<p><a href="https://bazel.build/" rel="nofollow">https://bazel.build/</a><p>Other than bazel you will have to start hacking away at dependency problems.<p>No inline imports, no circular imports. Imports all sorted at the top of your file. You will have to start enforcing good hygiene with linters.<p>You will need to create warnings against using the global scope.<p>You will need to construct the clients for all your dependencies in main()<p>You will need to discourage the use of calling non trivial functions in constructors. (this property largely encourages dependency injection).<p>There are exceptions to every rule, but if you are going to violate scope or not dependency inject, those things need to be done very mindfully.<p>As the structure of your code improves via good scoping and injected dependencies, it will become easier to change <i>and</i> easier to test.<p>You will have to devote some serious consideration to how to quarantine business logic from server code. Generally, your product developers shouldn't be doing much outside of defining their data and altering business logic from within a route. If the place where business logic is executed is commingled with how data-stores are manipulated, you're going to have a bad time. Likewise if the place business logic is executed is commingled with the presentation of it to customers, you're going to have a bad time.<p>Python does not have a culture of dependency injection because it's so easy to import antigravity and fly away. This makes writing tests hard and promotes spaghetti code. Lack of dependency injection (which means violating scoping) is the entropic force that makes codebases miserable as time increases.<p>Additionally, you will have to think hard about state. If you can't restart a process trivially, or balance traffic to a different machine trivially, you are going to make your operational people's lives hard. State belongs in state storage. Put it in an RDBMS, put it in redis, put it in memcached, put it in anything but a python processes memory (or disk). This means that any two requests should be able to be sent to any two machines. This is a deeply important property for scaling.<p>Lastly, if you do not have good answers for observability, in terms of time series data, log data, exception data, and event data (for observability only), you will have a bad time. These are generally the things it is ok to violate scope to use.