TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Redis at Bump: many roles, best practices, and lessons learned

94 pointsby jmintzover 14 years ago

7 comments

antirezover 14 years ago
Thank you for writing this article. As a way to show my appreciation I want to focus on the bad side of the matter.<p>The article mentions that with AOF persistence there is a problem about fsync. I'll try to go in further details here.<p>Basically when using Redis AOF you can select among three levels of fsync: fsync always that will call fsync after every command, before returning the OK message to the client. Bank-alike security that data was written on disk, but very very slow. Not what most users want.<p>Then there is 'fsync everysec' that just calls fsync every second. This is what most users want. And finally 'fsync never' that will let the OS decide when to flush things on disk. With Linux default config writing buffers on disk can be delayed up to 30 seconds.<p>So with fsync none, there are no problems, everything will be super fast, but durability is not great.<p>With fsync everysec, there is the problem that form time to time we need to fsync. Guess what? Even if we fsync in a different thread, write(2) will block anyway.<p>Usually this does not happen, as the disk is spare. But once you start compacting the log with the BGREWRITEAOF command, the disk I/O increases as there is a Redis child trying to perform the compaction, so the fsync() will start to be slow.<p>How to fix that? For now we introduced in Redis 2.2 an option that will not fsync the AOF file while writing IF there is a compaction in progress.<p>In the future we'll try to find even Linux-specific ways to fsync without blocking. We just want to say the kernel: please flush the current buffers, but even if you are doing so, new writes should go inside the write buffer, so don't try to delay new writes if the fsync in progress is not yet completed. This way we can just fsycn every second in another thread.<p>Another option is to write+fsync the AOF log in a different process, talking with the main process via a pipe. Guess what? The current setup at Bump is somewhat doing this already with the master-&#62;slave setup. But there should be no need to do this.<p>So surely things will improve.<p>About diskstore, this is I think better suited for a different use case, that is: big data, much bigger than RAM, but mostly reads, and need to restart the server without loading everything in memory. So I think Bump is already using Redis in the best way, just we need to improve the fsync() process.
评论 #2223253 未加载
评论 #2223201 未加载
rbransonover 14 years ago
I don't get what is so difficult about AMQP. These are clearly talented programmers, so what gives? Even if you're not a bank, simple features like message timeouts can make your infrastructure tremendously more resilient.
评论 #2222600 未加载
评论 #2223296 未加载
LiveTheDreamover 14 years ago
In January, Bump reported allocating a whopping 700GB of RAM for redis[1].<p>[1] <a href="http://devblog.bu.mp/haskell-at-bump" rel="nofollow">http://devblog.bu.mp/haskell-at-bump</a>
评论 #2222957 未加载
simonwover 14 years ago
I really like the idea of pushing log messages in to a redis list and then flushing them out to disk with another process.<p>I've often thought it would be useful to have a redis equivalent of MongoDB's capped collections, specifically to make things like recent activity logs easier to implement. At the moment you can simulate it with an rpush followed by an ltrim, but it would be nice if using two commands wasn't necessary.
评论 #2222899 未加载
评论 #2223077 未加载
ladon86over 14 years ago
OK, so I'm running mongodb on the same machine as redis.<p>I do have mongodb replicated across two other machines, but could you briefly shed light on what the problems between redis and mongo on a single box were?
评论 #2228486 未加载
sfphotoartsover 14 years ago
I was curious about the Redis sets used for social graph storage and using intersection of sets to find nodes in common. Would anyone have an idea about the complexity of this as both the number of nodes in the graph and the number of edges each node has?
评论 #2224879 未加载
moeover 14 years ago
Just a note about logging: It seems you're making it harder than it needs to be. Syslog supports remote-logging.