Hi HN ! Alex here. I'm excited to show you Asyncpal (<a href="https://github.com/pyrustic/asyncpal">https://github.com/pyrustic/asyncpal</a>), a Python library for parallelism [1] and preemptive concurrency [2] tailored for sporadic workloads [3].<p>I've been working on a private project where it would be convenient to have asynchronous versions of some operations without using `async/await` [4]. Besides `async/await`, which represents cooperative concurrency via the `asyncio` package, Python also provides preemptive concurrency through its `threading` package. Additionally, the `concurrent.futures` package builds upon the `threading` package to provide a thread pool [5], which is the design pattern my project needed for managing concurrent tasks.<p>Although a thread pool is the right tool for the problems it solves, its creation and usage involve the allocation of resources that must be properly released. For this reason, it is recommended to use a thread pool with a context manager [6] to ensure that resources are properly released once the pool executor has finished its tasks.<p>However, this strategy can introduce overhead in programs that sporadically submit tasks to a thread pool, as multiple pools may be created and destroyed throughout the execution of these programs.<p>Maintaining one or a few thread pools for the duration of a program can be an effective solution, assuming these thread pools can automatically shrink after workers have been idle for a short period defined by the programmer.<p>I developed Asyncpal to meet the requirements and extended it with processes to achieve parallelism. For more details about this project, please refer to the README. You can download the lightweight package on PyPI, try the examples provided in the README, or run the tests on your machine.<p>Let me know what you think about this project.<p>[1] <a href="https://en.wikipedia.org/wiki/Parallel_computing" rel="nofollow">https://en.wikipedia.org/wiki/Parallel_computing</a><p>[2] <a href="https://en.wikipedia.org/wiki/Concurrent_computing" rel="nofollow">https://en.wikipedia.org/wiki/Concurrent_computing</a><p>[3] <a href="https://www.cloudcomputingpatterns.org/unpredictable_workload/" rel="nofollow">https://www.cloudcomputingpatterns.org/unpredictable_workloa...</a> (related)<p>[4] <a href="https://en.wikipedia.org/wiki/Async/await" rel="nofollow">https://en.wikipedia.org/wiki/Async/await</a><p>[5] <a href="https://en.wikipedia.org/wiki/Thread_pool" rel="nofollow">https://en.wikipedia.org/wiki/Thread_pool</a><p>[6] <a href="https://superfastpython.com/threadpoolexecutor-context-manager/" rel="nofollow">https://superfastpython.com/threadpoolexecutor-context-manag...</a>
I was asked hours ago by a concurrency enthusiast, whose website has been a valuable source of information for me on the topic, to tell in a sentence what capability Asyncpal provides to users above the Python standard library (stdlib).<p>I found the question interesting because it goes straight to the point and calls for a concise answer. I believe the answer is missing from this 'Show HN'. Here is my response to the question:<p>Asyncpal unifies the stdlib (concurrent.futures + multiprocessing.pool) and provides true elastic pools (grow + shrink) with an intuitive interface.
Looks very well designed. As someone new to asynchronous programming in python, I have been using asyncio.to_thread in a task. Although the best way to manage things is to use a thread pool executor
that's cool. but i'm looking for a preemptive concurrency like golang in pyhton that schedules routines and get threads back when routine runs a blocking syscall. the hard part is detection of syscalls. is there any idea?