Are any of you maintaining any web scrapers in production?<p>I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.<p>Curious what applications people have in industry and what sorts of techniques people use for reliability.
excel web scraping[0] (vs. using python[1] and/or odbc/delimited files)<p>A few 2025 use cases [2],[3]:<p><pre><code> Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc
</code></pre>
----<p><pre><code> creative take on topic:
modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384
Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html
3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html
</code></pre>
----<p>[0] : excel scraping : <a href="https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk" rel="nofollow">https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk</a><p>[1] : python scraping : <a href="https://www.youtube.com/watch?v=Oo8-nEuDBkk" rel="nofollow">https://www.youtube.com/watch?v=Oo8-nEuDBkk</a><p>[2] : <a href="https://dataforest.ai/blog/top-web-scraping-use-cases" rel="nofollow">https://dataforest.ai/blog/top-web-scraping-use-cases</a><p>[3] : <a href="https://www.parsehub.com/blog/web-scraping-examples/" rel="nofollow">https://www.parsehub.com/blog/web-scraping-examples/</a>
Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.