TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

The magic of small databases

192 pointsby topcat31over 2 years ago

25 comments

btownover 2 years ago
Buried in here is a fascinating musing on &quot;Market-making Small Databases&quot; - &quot;Imagine a Substack for databases - an easy tool for creating, maintaining and publishing databases with the ability to restrict parts or all of it behind a pay wall. Pair it with the ability to send email updates to your audience about changes and additions...&quot; It&#x27;s worth a read in full in the original article.<p>One of my favorite small databases is <a href="https:&#x2F;&#x2F;hiregoats.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hiregoats.com&#x2F;</a> - it&#x27;s a simple site showing goat herds for rent (for clearing brush in a sustainable way, etc.), monetized with at $35 listing fee and nothing else. There&#x27;s no e-commerce, no attempt to insert the site into the transaction or funds flow, no bells and whistles. Certainly this doesn&#x27;t scale to other niches where suppliers are less incentivized to pay a listing fee, but I&#x27;d love to see this kind of thing be more common, and incentivize people to curate.
评论 #34561001 未加载
评论 #34565058 未加载
dmjeover 2 years ago
I run a little agency in the UK who works with museums to help them with digital. A large part of this is getting collections online.<p>Some years ago we commissioned a developer to make CultureObject[0], a free and open source WordPress plugin to make it easier to ingest collections data for display on the web. At the heart it&#x27;s a glorified data importer, and many people just use the CSV mode to sync and import collections data.<p>It requires some dev effort - we&#x27;ve built an add-on which makes this easier but there&#x27;s no denying that search, faceting and display needs knowledge of wordpress development.<p>Three years ago we then launched The Museum Platform[1] which is a more SaaS based model - we take away the need for dev skills and ask clients to just send us a CSV and any related media and we do the hard work. It&#x27;s WordPress again but a modified version where we also facilitate storytelling and narrative around the ingested collections.<p>The interesting thing about this journey is that the requirement to &quot;get a collection online&quot; is apparently and theoretically easy. But the reality is it gets hard quite quickly as the need for search &#x2F; filtering appears, and it gets harder still as scale comes into it. 1000 records is fine. 100,000 gets quite a bit harder.<p>There are also many subtleties - particularly with museum collections. &quot;Location&quot; of a record could be where it was collected, or where it is now, or where it&#x27;s on display. Relational stuff is hard, as are taxonomies and authority terms. It&#x27;s hard to generalise and it&#x27;s hard to scale.<p>[0] <a href="https:&#x2F;&#x2F;cultureobject.co.uk&#x2F;" rel="nofollow">https:&#x2F;&#x2F;cultureobject.co.uk&#x2F;</a> [1] <a href="https:&#x2F;&#x2F;themuseumplatform.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;themuseumplatform.com&#x2F;</a>
评论 #34566382 未加载
评论 #34563544 未加载
评论 #34567730 未加载
评论 #34564412 未加载
评论 #34566212 未加载
breckover 2 years ago
I&#x27;m going to plug our related project: TreeBase. It&#x27;s the public domain software that powers PLDB.com (a Programming Language DataBase).<p>It&#x27;s very simple. If your small database was about cars, your structure might look something like this:<p><pre><code> database&#x2F; grammar&#x2F; engine.grammar interior.grammar things&#x2F; model3.car camry.car </code></pre> The `grammar` files are written in a Tree Language called Grammar. Those are your schema files. You basically create a new syntax-free plain text &quot;language&quot; for storing your data, in this case 1 &quot;car&quot; file per model of car.<p>It was a pipedream of mine until the M1&#x27;s came out. Those changed everything, because then it became fast enough to actually do it.<p>We have a new release coming out soon with a new query language that will change everything. Here is the source code: <a href="https:&#x2F;&#x2F;github.com&#x2F;breck7&#x2F;jtree&#x2F;tree&#x2F;main&#x2F;treeBase">https:&#x2F;&#x2F;github.com&#x2F;breck7&#x2F;jtree&#x2F;tree&#x2F;main&#x2F;treeBase</a>
LunarAuroraover 2 years ago
There are categories of “Nocode” online services that could work, more or less, as small databases. Some are already cited in the article:<p>- DBs platforms (Best for more advanced DB) : Airtable, getgrist.com<p>- wikis+DB platforms (Best for building a site around the DB) : notion.so, coda.io<p>- Airtable&#x2F;GSheet publishing (Best for simple&#x2F;custom UI) : glideapps.com, siteoly.com<p>- Bookmarks&#x2F;Collections (Best for links&#x2F;References) : Zotero (online groups), are.na<p>- List sharing (Best for open collaboration?) : listium.com, (ranker.com ?)<p>- BI platforms (Best for advanced filters&#x2F;charts) : polymersearch.com, Google Data Studio<p>- Data Set Hosting (Best for downloading?) : data.world, kaggle.com<p>All these allow publishing, and some collaboration
评论 #34561393 未加载
xnxover 2 years ago
&quot;Publishing documents to the web is a well-served use case but publishing small indexes, databases and collections to the web is still an incredibly frustrating and under-served use case. Here I outline why I think it matters and a variety of approaches to solving it.&quot;<p>Amen. I&#x27;m surprised the post doesn&#x27;t mention sqlite3 WASM&#x2F;JS (<a href="https:&#x2F;&#x2F;sqlite.org&#x2F;wasm&#x2F;doc&#x2F;trunk&#x2F;about.md" rel="nofollow">https:&#x2F;&#x2F;sqlite.org&#x2F;wasm&#x2F;doc&#x2F;trunk&#x2F;about.md</a>). That, paired with an easy-to-use faceting library, would go a long way.
ZephyrBluover 2 years ago
I love this. I&#x27;ve been thinking about something similar lately. There are so few good indexes and search engines for niche collections of data.<p>Imagine if there was a niche search engine for everything, and the search engine was customized for that niche.<p>I think the main problems here are:<p>- Data format and ingestion - Domain-specific indexing&#x2F;relevance<p>Most data is super messy and it not accessible through nice APIs, which presents a problem. You might need custom ingestion for each niche and it&#x27;s pretty likely you&#x27;ll need some rules to standardize data from multiple sources, neither of which seems easy to generalize and automate because they&#x27;re very domain-specific.<p>The other part to this is indexing&#x2F;relevance so the search feels good to use. Some fields are obviously going to be more important than others and people are going to want to utilize search for things that are to predict ahead of time.<p>To use the authors example of artists in Brooklyn, people might want to search for artists near them. Now you have to gather location data, format it, ingest it, index it and add it to the search UI.<p>The fact that adding another field to index on is a vertical integration adds a lot of overhead.<p>All of this stuff in isolation is not difficult, but when you put it together it becomes quite a lot of work that generally isn&#x27;t easily scalable.
itsmemattchungover 2 years ago
Reminds of Amazon EBS and a white paper describing the philosophy of deploying millions of tiny databases:<p><a href="https:&#x2F;&#x2F;assets.amazon.science&#x2F;c4&#x2F;11&#x2F;de2606884b63bf4d95190a3c2390&#x2F;millions-of-tiny-databases.pdf" rel="nofollow">https:&#x2F;&#x2F;assets.amazon.science&#x2F;c4&#x2F;11&#x2F;de2606884b63bf4d95190a3c...</a>
zokierover 2 years ago
Personally I find the whole dBase etc non-SQL kinda-graphical database systems interesting historical software branch that feels mostly died out these days. Access probably did quite a lot of damage here, killing out competitors before mostly succumbing itself.
评论 #34560971 未加载
评论 #34561042 未加载
simongrayover 2 years ago
This post is an exercise in describing the motivation and features of the Semantic Web seemingly without realising the tech stack already exists.
评论 #34561289 未加载
moehmover 2 years ago
For what it&#x27;s worth, here is my &quot;small database&quot; attempt, a structured list of worthwhile Wikipedia articles to read.<p><a href="https:&#x2F;&#x2F;www.mostdiscussed.com" rel="nofollow">https:&#x2F;&#x2F;www.mostdiscussed.com</a>
overgardover 2 years ago
People would love this for sports. There&#x27;s so much interesting data locked up in proprietary databases
roncesvallesover 2 years ago
I&#x27;m aware that this may sound dismissive but the solution that the author of the OP is looking for <i>is</i> the World Wide Web itself.<p>The &quot;small database&quot; in question is, well, an HTML page. It can be shared and passed around by selecting the portions of it that you need and pressing Ctrl+C&#x2F;Ctrl+V. Search is accomplished by the browser using Ctrl+F. Collaboration can take many forms - wikis, comments, forums, live editing. Links between databases are what URL links are. The database that OP is looking for is a page of text (for unstructured data) or somewhat structured solutions like CSV, JSON, or YAML.<p>Now, yes, there are certain participants on the WWW who make poor web design choices that cause agreed-upon functionality to break. E.g. unnecessary pagination or accordions breaking Ctrl+F, not offering data for download, not having useful URL paths etc.
Zababaover 2 years ago
I like the idea, but I think one issue is that the database is the easy part. If I look again at the list of requirements, most are not about the database but about how to put data from external source in the database, how to edit the database, and how to publish it. To me this sounds like an interface problem. But since the whole point is small, specialized collections, interfaces have to be specialized too. That means no single tool that can offer a solution. Maybe it&#x27;s an issue of definition, I call a database something like MySQL or SQLite or even a CSV file, while for the author it&#x27;s the finished product, the database about &lt;stuff&gt; and the tools that are adapted to &lt;stuff&gt;.<p>Substack is an interesting example. It&#x27;s great for written content with a few images, which mostly looks the same everywhere. But it lacks great customisation features that I think a database would need, because that stuff is hard to do.<p>If I had to propose a solution, it would be this: if you want to do a small database, do it. Experimentation in the cyberspace is very cheap. These days you have lots of resources for everything online. It can be intimidating, and can lead to analysis paralysis. I&#x27;m supposed to be a professional developer and still struggle with that. But one thing that has helped me a lot recently is to try stuff, see if it works, if it fails, ask questions (to either real people or ChatGPT&#x2F;Copilot, Copilot is especially valuable to get in a &quot;just keep writing, editing comes later&quot; mood). It&#x27;s not always fun, in fact it can be quite frustrating, but that&#x27;s how things are.<p>In the end, this is about decentralisation and you can&#x27;t have proper decentralisation if you don&#x27;t also decentralise the skills, the know-how. For example, there has been a lot of talk about Mastodon as a decentralised alternative to Twitter. And it is one. But if you simply go from being a user on Twitter to being a user on Mastodon, well you don&#x27;t regain much control. On the other hand if you try running a small instance, even just a local instance to see how it works, or maybe add a few feature to your preferred client (it can be code, but it could also be helping translation, or maybe a color scheme (you wouldn&#x27;t believe how many color scheme are barely usable when you&#x27;re colorblind)), well then you start being in control.
dgudkovover 2 years ago
Small databases aren&#x27;t popular because Excel spreadsheets already occupy that niche. A small database doesn&#x27;t have to be normalized. Because it&#x27;s small, it can be denormalized into a flat table that can be conveniently handled in Excel.
评论 #34563264 未加载
topcat31over 2 years ago
Hey OP here, just wanted to say thanks for all the comments (goats and all). There&#x27;s lots I still need to learn about (actual) databases as a hobby developer...<p>In the meantime I&#x27;ve made a big update to the Airtable with links to tools, examples and further reading:<p><a href="https:&#x2F;&#x2F;airtable.com&#x2F;shrYY94GrqVB4HUsi&#x2F;tblHPrdomiPbLpod6&#x2F;viwxizssDJMsGqhg9?backgroundColor=green&amp;blocks=hide" rel="nofollow">https:&#x2F;&#x2F;airtable.com&#x2F;shrYY94GrqVB4HUsi&#x2F;tblHPrdomiPbLpod6&#x2F;viw...</a>
marniewebbover 2 years ago
H2O — <a href="https:&#x2F;&#x2F;h2o.law.harvard.edu&#x2F;" rel="nofollow">https:&#x2F;&#x2F;h2o.law.harvard.edu&#x2F;</a> — is a now-defunct collaborative syllabus project from Harvard that gets at a lot of this I think. It’s basically a list maker with a lot of additional capabilities. While it’s made for small list of things it’s easy to imagine this is a piece of the solution.
082349872349872over 2 years ago
A search for &quot;filemaker&quot; reveals that Claris is still in business; I&#x27;d hope they&#x27;d have <i>something</i> that might address this need?
hardwaresoftonover 2 years ago
Weirdly enough I haven’t seen too much mention of CMSes — them plus&#x2F;minus spreadsheet like tools are almost surely the way to handle this kind of use case.<p>What’s missing is the added search + UI capabilities.<p>I think about saas ideas a lot and this is actually quite a common one (though I’m generally thinking of a specific niche) —- enabling people to craft and expose datasets would surely be a great startup.
jerryuover 2 years ago
Having a small database is especially useful when collaborating on data strategy. I have seen some database diagrams with 1000s of tables and it is hard to make sense of it using ERD tools.<p>Even with advanced views offered by tools like ERDLab.io it is a pain in the ass to collaborate on large schemas at various stages of development.
cavisneover 2 years ago
I feel like this is getting really close. GPT is create at writing sql queries from text and turning a blob of semi structured data into an sql schema.<p>We just need to somehow tie it together so anyone can explain their use case, and show an example of the data in plain english, then lock in a schema and feed everything in.
aabbcc1241over 2 years ago
For collection of links with short description for projects&#x2F;services, there are many awesome list on github.<p>For more complex data to be shared, maybe it can be csv&#x2F;md&#x2F;mdx shared over git as well?<p>It can have stable url and be searchable from github, search engines, and 3rd indicies
vaporupover 2 years ago
<a href="https:&#x2F;&#x2F;zed.brimdata.io" rel="nofollow">https:&#x2F;&#x2F;zed.brimdata.io</a>
maphewover 2 years ago
Makes me think of what something like Datasette fused with Fossil SCM could accomplish.
Trayja-Peterover 2 years ago
&quot;I want to empower more individuals to publish, maintain and collaborate on small indexes. To build a million tiny libraries, community databases, weird collections and indie indexes.&quot;<p>Funnily enough, a friend and I have been building <a href="https:&#x2F;&#x2F;Trayja.com" rel="nofollow">https:&#x2F;&#x2F;Trayja.com</a>, a tool which does this exact thing, with a focus on the &quot;community&quot; aspect. There&#x27;s a huge amount of wisdom in communities, whose value could be multiplied if it would be aggregated in a structured, indexable, searchable way. This article articulated so much of what I&#x27;ve been trying to explain about my project.
LAC-Techover 2 years ago
With how fast computers are now, they can work well for small businesses too.