TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Rearchitecting: Redis to SQLite

360 点作者 thecodemonkey8 个月前

27 条评论

simonw8 个月前
I’m really interested in this model where each application server has a copy of a SQLite database file which is then replaced on a scheduled basis.<p>Here it’s being used for web application firewall rules.<p>Another place I’ve thought about using this is feature flag configuration. Feature flags can be checked dozens of times per request and often need the kind of queries (user is a member of group A and has an IP located in country B) which could be well served by a local SQLite - and feature flags have a tolerance for updates taking a few seconds (or longer) to roll out.
评论 #41647295 未加载
评论 #41648480 未加载
评论 #41647805 未加载
评论 #41658940 未加载
评论 #41646879 未加载
评论 #41648396 未加载
评论 #41649120 未加载
评论 #41652675 未加载
评论 #41661028 未加载
评论 #41651883 未加载
评论 #41653299 未加载
评论 #41647988 未加载
评论 #41651938 未加载
评论 #41649177 未加载
评论 #41648210 未加载
vchynarov8 个月前
Apart from network latency, one of the behaviours I&#x27;ve seen with Redis is that reads&#x2F;write latencies are fairly linearly proportional to the amount of keys queried - which seems to be shown in your chart as well.<p>We had a different problem, where our monolithic app used both Postgres &#x2F; Redis for different use cases and worked relatively well. However - it was a lot easier to shove new functionality in the shared Redis cluster. Because Redis is single-threaded, one inconsiderate feature that does bulk reads (100K+ keys) may start to slow down other things. One of the guidelines I proposed was that Redis is really good when we&#x27;re reading&#x2F;writing a key, or small fixed-cardinality set of keys at a time, because we have a lot of random things using Redis (things like locks and rate limits on popular endpoints, etc).<p>However, in your case, I&#x27;m guessing Redis shines in the case of a naive single-key (IP address) lookup, but also doesn&#x27;t do well with more complicated reads (representing your range query representation?). Cool write up overall, I don&#x27;t have a deeper understanding of how SQLite performs so well when compared to a local Redis instance, so that was unexpected and interesting to observe.
评论 #41654854 未加载
aquilaFiera8 个月前
Somewhat related: for the Neon internal hackathon a few weeks ago I wrote a little Node.js server that turns Redis&#x27;s wire protocol (RESP) into Postgres queries. Very fun hack project: <a href="https:&#x2F;&#x2F;github.com&#x2F;btholt&#x2F;redis-to-postgres">https:&#x2F;&#x2F;github.com&#x2F;btholt&#x2F;redis-to-postgres</a>
matharmin8 个月前
It sounds like a niche use case where SQLite does work quite well server-side without needing any replication, since the database is read-only.<p>Other alternatives may use static files loaded in-memory, but I&#x27;m guessing the data is more than you&#x27;d want to keep in memory in this case, making SQLite a nice alternative.
评论 #41646597 未加载
评论 #41648342 未加载
favorited8 个月前
&gt; Further, when we exhibited at RailsWorld 2023, there was a definite &quot;blood in the water&quot; vibe regarding Redis and the assumption that you&#x27;d automatically need a Redis server running alongside your Rails application.<p>I&#x27;ve only worked on one production Rails application in my career (and it did use Redis!), so I&#x27;m way out of the loop – is the ecosystem turning against Redis from a business perspective (I know there have been some license changes), or is it a YAGNI situation, or something else?<p>IIRC we used it mainly with Rescue to schedule asynchronous jobs like indexing, transcoding, etc., but it seemed like a neat tool at the time.
评论 #41651180 未加载
评论 #41651108 未加载
评论 #41658822 未加载
评论 #41660893 未加载
macspoofing8 个月前
&gt;While Redis is &quot;fast&quot; in comparison to traditional RDBMS, it&#x27;s still a database that you have to manage connections, memory, processes, etc., which introduces more brittleness into the stack (the opposite of what we&#x27;re trying to achieve).<p>Every database, Relational or Nonrelational, requires approximately the same level of management and maintenance when you start dealing with non-toy levels of transactions.<p>The &quot;Fast&quot; part is a little funny. If you don&#x27;t care about joins, then row inserts and retrievals are pretty damn fast too =)
评论 #41648376 未加载
评论 #41647800 未加载
评论 #41685276 未加载
评论 #41647459 未加载
keybits8 个月前
People reading this might be interested in Redka - Redis re-implemented with SQLite in Go: <a href="https:&#x2F;&#x2F;github.com&#x2F;nalgeon&#x2F;redka">https:&#x2F;&#x2F;github.com&#x2F;nalgeon&#x2F;redka</a>
评论 #41648460 未加载
评论 #41651609 未加载
avovana8 个月前
Could you clarify, please Redis usage?<p>v1 1) In v1 they had waf and redis on the same server 2) Client went to the admin panel to set new rules 3) Rules went to redis that is on the same server with admin panel 4) Thanks to redis internal synchronization mechanism rules were updated to all of the redises(that are stand locally with waf all over the globe) 5) When new request come to some waf, waf verified request&#x2F;ip with updated redis rules Do I understand v1 correctly? Redis infrastructure was used to spread new rules by itself?<p>v2: 1) They deleted the redis cluster 2) Every waf server now has sqlite db 3) They made some synchronization mechanism to spread new rules from admin panel to every server that contains waf and sqlite 4) When a new request comes to some waf, waf verifies request&#x2F;ip with updated sqlite rules. And that is very fast!<p>That is the case?
tony-allan8 个月前
Best quote:<p>&quot;SQLite does not compete with client&#x2F;server databases. SQLite competes with fopen().&quot;
theamk8 个月前
The dataset is 1.2 million entries, which looks big, but really is not that much.<p>If this is uncompressed IPv4 addresses, it&#x27;s just 4.8 MB; and with some trival compression (like a 2-level trie), it could be about 2x smaller. Even if it&#x27;s uncompressed IPv6, that&#x27;s still just 32 megabytes.<p>Does Ruby support mmap? If yes, I&#x27;d suggest direct IP list. Lots of fun to write, big speedup over sqlite, and zero startup time.
codingbot30008 个月前
It&#x27;s posts like this explaining architecture decisions in detail I am reading HN for. Thank you!
评论 #41649428 未加载
prirun8 个月前
Might want to check into this to do your SQLite db copies:<p><a href="https:&#x2F;&#x2F;www.sqlite.org&#x2F;draft&#x2F;rsync.html" rel="nofollow">https:&#x2F;&#x2F;www.sqlite.org&#x2F;draft&#x2F;rsync.html</a>
评论 #41654074 未加载
doubleorseven8 个月前
&gt; Benchmarking is a dark art of deceiving yourself with highly precise numbers<p>.
dangoodmanUT8 个月前
I have a hard time believing that Redis local was beat by SQLite local unless the workload was poorly fit for Redis structures, or the integration code wasn&#x27;t well written.<p>But always happy to see a discovery of a better solution. I agree removing the network is a win.
评论 #41648109 未加载
评论 #41646952 未加载
评论 #41652905 未加载
ten138 个月前
Nice post! I’m curious how the SQLite-per-instance model works for rate-limiting in the scale-out scenario. I took a cursory glance at the docs but nothing jumped out at me about how it works.
评论 #41646580 未加载
nikisweeting8 个月前
I really wish there were a compatibility layer that could sit on top of SQLite and make it pretend to be redis, so we could switch more things to use SQLite. It doesn&#x27;t even need to satisfy all the distributed systems guarantess or even implement proper pub&#x2F;sub, it could just do everything with polling and a single event loop. It would be great for smaller projects that want to run something like celery or any app that depends on redis without needing to install redis.
评论 #41654477 未加载
评论 #41651595 未加载
rini178 个月前
If you need writes, can just use second sqlite database.
评论 #41647483 未加载
masfoobar8 个月前
NICE!<p>I have not used Redis myself, but have been using Sqlite more and more over the years.. and found a perfect application I wrote using Sqlite under the hood.<p>Powerful and convienient database system!
gwbas1c8 个月前
How large is the SQLite database you&#x27;re syncing?<p>Is it even &quot;worth&quot; using SQLite at this point? What about a configuration file, and straight-up code that works with in-memory data structures?
评论 #41647939 未加载
评论 #41648637 未加载
ragu4u8 个月前
So is the sqlite file on disk or in memory somehow?
评论 #41647393 未加载
评论 #41648329 未加载
tmaier8 个月前
I visited this site from safari on iOS while being in a Marriott hotel. I am blocked. So the WAF works.
lilatree8 个月前
I wish there was a repository with lots of posts like this one. Super useful to learn from!
jszymborski8 个月前
A bit strange they replaced Redis with SQLite rather than LMDB or RocksDB which are key-value stores
avinassh8 个月前
Is the benchmark code available somewhere &#x2F; open source?
justinclift8 个月前
Wonder if they had indexes on their SQLite tables?<p>Not seeing a mention of that in the article.
评论 #41647538 未加载
singpolyma38 个月前
&gt; Even if the SQLite performance was significantly worse (like 2x worse) in the benchmark, it would still probably be faster in the &quot;real world&quot; because of network latency, even to a Redis that was in the same data center&#x2F;region<p>... Why not run redis on localhost?
评论 #41660417 未加载
tiffanyh8 个月前
FoundationDB<p>Isn’t “redis to sqlite” effectively what foundationDB?<p><a href="https:&#x2F;&#x2F;www.foundationdb.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.foundationdb.org&#x2F;</a>