I want to clarify a couple of things. I'm not saying that Paul Graham invented this pattern. Actually after he mentioned it, I remembered a friend of my father to implement exactly that in QUICKBASIC in the late 80s :-) The point is that maybe the Redis design was already inside me, but I needed a trigger: I often think of good things after being triggered. And smart people are more likely to tell about good ideas, old and new. That was the point. Similarly I believe there are a lot of simple fundamental ideas that can be re-applied to today's technology, as the landscape changes many things become relevant again.
Man, some of the comments on here are really disheartening. What's the deal with trying to humble people with long-winded, tangential counterpoints, gotchas, and condescending questions? Constantly proving one's intellect seems to be a prevailing theme on HN and I'm not seeing how it adds to the quality of the content. I'm sure someone will unearth the irony in this and let me know soon enough.
That's exactly how HyperCard works, and we all know what inspired HyperCard!<p><a href="https://www.mondo2000.com/2018/06/18/the-inspiration-for-hypercard/" rel="nofollow">https://www.mondo2000.com/2018/06/18/the-inspiration-for-hyp...</a><p><a href="https://twit.tv/shows/triangulation/episodes/247?autostart=false" rel="nofollow">https://twit.tv/shows/triangulation/episodes/247?autostart=f...</a>
Hmm, I thought this pattern was really common? That is, appending everything to a file and reading back from the file when there's a reboot. I constantly use it when a database (or redis for that matter) is simply overkill for my use case.<p>Here's a 34 line implementation I use on a node production system. It writes struct-like JavaScript objects that represents events to disk. When reading it back I do a fold (or .reduce) to build the state.<p>And yes –– it could be way smarter (writing to memory <i>and</i> disk), but YAGNI has been working out pretty well so far.<p><pre><code> class EventStore {
constructor(file) {
this.file = file;
this.cache = null;
}
async appendEvent(event) {
// Purge the cache for entries since we mutated the store.
this.cache = null;
return new Promise((resolve, reject) => {
createWriteStream(this.file, { flags: 'a' })
.on('error', reject)
.on('close', resolve)
.end(`${JSON.stringify(event)}\n`);
});
}
async readEvents() {
if (this.cache !== null) {
return this.cache;
}
try {
const data = await readFile(this.file, 'utf-8');
const lines = data.split('\n').filter(line => line);
const events = lines.map(line => JSON.parse(line));
this.cache = events;
return events;
} catch (error) {
return [];
}
}
}</code></pre>
To offer a slightly alternative perpective on this, I actually think this type of "article"/("listicle"/"tweetacle"?) can have negative effects. In my mind, it lends credence to the very toxic notion of "value of ideas" over "value of execution".<p>The former is something that has all sorts of knock-on effects: backward IP laws "protecting" ideas, perverse incentives within large corporations with outspoken "idea men" being promoted ahead of doers, non-technical founders with "high-potential ideas" sucking up investment and expecting to execute with technical hires on untested theories.<p>I'm not saying any of the above applies in this case of course, but the fact is that it is you who built Redis, not pg, nor many others who've had similar ideas, and I think writing the above tweet thread lends undue weight to many of above negative trends in our industry (and also in general in recorded history of IP/invention-credit battles).
“System prevalence[1] is a simple software architectural pattern that combines system images (snapshots) and transaction journaling to provide speed, performance scalability, transparent persistence and transparent live mirroring of computer system state.“ — <a href="https://en.m.wikipedia.org/wiki/System_prevalence" rel="nofollow">https://en.m.wikipedia.org/wiki/System_prevalence</a>
PG was also involved in the inception of Reddit: It was PG who gave Alexis and Steve the idea to make something like reddit, and also gave them the tagline "the front page of the internet".[0]
PG had vetoed their initial idea to create a food-delivery app and then called them back and asked them to come up with something new.
[0]: <a href="https://www.youtube.com/watch?v=5rZ8f3Bx6Po" rel="nofollow">https://www.youtube.com/watch?v=5rZ8f3Bx6Po</a>
<i>When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases themselves could be that way instead of using the programming language data structures without a networked API.</i><p>That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.
Of course Erlang has had ETS (and of course you can use gen_servers of various kinds for this) built in forever. Redis is fantastic but I think there have been many examples of prior art before this tweet!
Is this really a brilliant idea though?<p>Everybody knows RAM became cheaper cheaper, while mechanical disk can't be made faster, and SSD have reliability limits.<p>It seems entirely logical to expect databases to work on RAM first and then commit on disk for a large performance improvement.