TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Ask HN: How would you build a custom web scraper?

2 pointsby yashbhatnagarover 2 years ago
I'm trying to build a web scraper that can scrape similar websites. For example, if I wanted to scrape 10 different fast food sites and return the locations the stores are in, it would be able to identify different elements and do so. Any ideas on how to implement this?

2 comments

sharemywinover 2 years ago
1. crawl pages looking for more pages on the site<p>2. scrape text<p>3. find address using something like this:<p><a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;9397485&#x2F;regex-street-address-match" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;9397485&#x2F;regex-street-add...</a>
JoeyBananasover 2 years ago
Use Scrapy to get at least some measure of maintainability.