TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Ask HN: How do you store large amounts of structured data?

3 pointsby bobblywobblesover 3 years ago
I&#x27;m looking to save large amounts of structured data - think coupons, recipes, ads, prices, etc. I&#x27;d like to be able to not have to worry about hard drives, so one thought was to save this in Github. However, the value proposition behind putting this in a SQL server is higher.<p>Have you stored large amounts of data before, how did you end up saving it?

4 comments

mikewarotover 3 years ago
I have a mess of folders with random stuff in it. Don&#x27;t do that.<p>However, if you tend towards <i>curation</i> more than me, you have categories, it might work for you.<p>I have a WikidPad personal wiki, in there are various things I&#x27;ve written over the past few decades, including the always elusive key to my WiFi.<p>My photos (600 GB) sit in folders D:\masterarchive\yyyy\yyyymmdd\ I never edit the originals, always save to a new file name when opening them in GIMP, etc. I have two short python scripts today.py and yesterday.py that make and open a new folder for me, depending on when I transfer the files from the SD card from my DSLR. You could make a script do it for you by reading the metadata from your smartphone photos.<p>SQL is a pain in the ass if you don&#x27;t match the structures you set up initially. It&#x27;s going to always be running, and will slow down a laptop or notebook boot time considerably. However, they do work well for <i>tightly</i> structured data.
dustedover 3 years ago
Github uses harddrives, and I believe SQL servers do as well, and you will (should) worry about them by proxy instead of directly. If you believe someone else (github, or whatevercloud) is going to care more about your data on their drives, than you about your data on your drives, well, best of luck :)<p>Still, it might make sense to use a database to store your documents, depending on your situations, like, is a receipt a text document or an image?<p>What are the access patterns you expect? Is it write and forget, or will you access it once a year, or once each hour? how searchable do you need it? What is large amounts? Terabytes? Petabytes?<p>A nice directory structure could still get you very far, depending on your needs, maybe even one with accompanying database for use as a searchable index to find the document (file) you need.
scary-sizeover 3 years ago
Define &quot;large&quot;. Also define your access pattern.
PaulHouleover 3 years ago
Arangodb