TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Against unnecessary databases: Parse, don't normalize

2 pointsby pcr910303almost 3 years ago

1 comment

vivegialmost 3 years ago
This makes a lot of sense. In the past when I built scrapers&#x2F;aggregators, it made sense to separate out the scrape part and the aggregate part.<p>1. It decoupled the steps and allowed for concurrency of the scraping and aggregation tasks. 2. It helped with debugging the tiny inconsistencies across sites and iterating on the aggregation. 3. As a side-benefit, if I needed to aggregate more data elements from the same sources, I could just re-run the aggregate step.