Ways to shoot yourself in the foot with Redis

My team manages a handful of clusters at work and I wrote on an internal redis client proxy (it&#x27;s on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache):<p>- Connection pooling &#x2F; pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it&#x27;s important developers understand the behavior of the client library they are using. Someone suggested using Envoy as sidecar proxy, I personally wouldn&#x27;t after our experience with it with redis but it&#x27;s an easy option. - Avoid changing the cluster topology if the CPU load is over 40%. This is primarily in case of unplanned failures during a change. - If something goes wrong shed load application side as quick as possible because Redis won&#x27;t recover if it&#x27;s being hammered. You&#x27;ll need to either have feature flags of be able to scale down your application. - Having replicas won&#x27;t protect you from data loss so don&#x27;t treat it as a source of truth. Also, don&#x27;t rely on consistency in clustered mode. - Remember Redis is single threaded so an 8xl isn&#x27;t going to be super useful with all those unused cores.<p>Things we have alarms on by default: - Engine utilization - Anomalies in replication lag - Network throughput (relative to throughput of the underlying EC2 instance) - Bytes used for cache - Swap usage (this is the oh shit alarm)

Another one: don&#x27;t use distributed locks using Redis (Redlock) as if they were just another mutex.<p>Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis&#x27; TTL if I remember correctly) - other instances will assume the lock is released after N seconds, to make sure an app instance which died does not leave the lock in the acquired state forever. So one day responses from the third party API started taking more time than Redlock&#x27;s timeout. Other app instances were assuming the lock was released and basically started accessing the API simultaneously without any synchronization. Data corruption ensued.

&gt; I wrote a basic session cache using GET, which fell back to a database query and SET to populate the cache in the event of a miss. Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. Increased traffic, combined with a slow query in Postgres, caused this arrangement to effectively DOS our Redis connection pool for minutes at a time.<p>This has nothing to do with the redis server. This is bad application code monopolizing a single connection waiting for an unrelated operation. A stateless request &#x2F; response to interact with redis for the individual operations does not hold any such locks.

Don&#x27;t expose your redis to the internet (please!). Don&#x27;t whitelist large swathes of your cloud&#x2F;hosting provider&#x27;s subnets either. Of course redis isn&#x27;t special, mongo, elastic, docker, k8s,etc... even if it is a testing server and you will never put important data on it.

&gt; One common mistake is serialising objects to JSON strings before storing them in Redis. This works for reading and writing objects as atomic units but is inefficient for reading or updating individual properties within an object<p>I would love to see some numbers on this. My intuition says there are probably some workloads where JSON strings are better and some where one key per property is better.

I had a jr dev connect and typed &#x27;flushall&#x27; because he thought it would refresh the dataset to disk.<p>thankfully it was on a staging env, I think he&#x27;s at google now.

I&#x27;ve found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you&#x27;ll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn&#x27;t the case.<p>I&#x27;ve noticed this particularly with Ruby where the official gem has cluster and sentinel support, but many other gems that depend on Redis expose their own abstraction for configuring it and it isn&#x27;t compatible with the official package.<p>Of course, I think that running Redis in clustered mode is actually just another way to shoot yourself in the foot, especially if a standalone instance isn&#x27;t causing you any trouble, as you can easily run into problems with resharding or poorly distributing the keyspace. Maybe just try out Sentinal for HA and failover support if you want some resilience.

Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.

Redis is great, it&#x27;s a great piece of software. One way I shot myself in the foot (kinda) with it: used it with a 1:N query fanout pattern. I.e., issued N queries to Redis for 1 incoming query to my service. My service by design needs to do N queries (it&#x27;s a long story). But Redis is not really designed to be used like this and I was putting it under very high load. I swapped it out with an SQLite cache recently and got rid of the errors that would pop up from putting extreme stress on the Redis server.

&gt; Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation.<p>What? Was this inside a MULTI (transaction) or something? This isn&#x27;t a flaw of Redis being single-threaded. Honestly all of these &quot;footguns&quot; sound like amateur programmer mistakes and have zero to do with Redis.

I have been using Redis for a long time and one of the things I love about it, is how difficult it is to shoot yourself in the foot with it. From the first use, after briefly reading some basic tips on what not to do, it was ridiculously simple to just get to work with it. I&#x27;ve never once run into a security or performance issue with it.

Change the default `stop-writes-on-bgsave-error` to &quot;no&quot; or you&#x27;re asking for trouble... a ticking time bomb.