estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.<p>It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.<p>estela has been recently published as OSS under the MIT license:<p><a href="https://github.com/bitmakerla/estela" rel="nofollow">https://github.com/bitmakerla/estela</a><p>More details about it can be found in the release blog post and the official documentation:<p><a href="https://bitmaker.la/blog/2022/06/24/estela-oss-release.html" rel="nofollow">https://bitmaker.la/blog/2022/06/24/estela-oss-release.html</a><p><a href="https://estela.bitmaker.la/docs/" rel="nofollow">https://estela.bitmaker.la/docs/</a><p>estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.<p>All kinds of feedback and contributions are welcome!<p>Disclaimer: I'm part of the development team behind estela :-)