Find non-expiring keys in Redis

We noticed that one of our production servers had an ever increasing memory usage increase. The memory increase was slow though.

It looked like some of the keys didn’t had the TTL in our redis servers. We wanted to find out how.

One of the standard ways is running the following command:

redis-cli keys  "*" | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq  -1 ]; then echo "$LINE"; fi; done;

However, running this command can have disastrous consequences on production server as keys is a blocking command and if you have millions of keys, it can make other requests to your redis server block.

There is another one SCAN which doesn’t block

redis-cli --scan | while read LINE ; do TTL=`redis-cli ttl "$LINE"`; if [ $TTL -eq  -1 ]; then echo "$LINE"; fi; done;

However, running these commands was inefficient for us we had millions of keys to analyze.

The solution came through Redis RDB tools

This tool can dump the keys and related information such as TTL, value, types in a csv file.

It can be installed through

pip install rdbtools python-lzf

And then run the following python script to point it to your rdb file. (rdb file can be generated by running bgsave command if the persistence is off.)

from rdbtools import RdbParser, RdbCallback
from rdbtools.encodehelpers import bytes_to_unicode

class MyCallback(RdbCallback):
    ''' Simple example to show how callback works.
        See RdbCallback for all available callback methods.
        See JsonCallback for a concrete example
    '''

    def __init__(self):
        super(MyCallback, self).__init__(string_escape=None)

    def encode_key(self, key):
        return bytes_to_unicode(key, self._escape, skip_printable=True)

    def encode_value(self, val):
        return bytes_to_unicode(val, self._escape)

    def set(self, key, value, expiry, info):
        print('%s = %s' % (self.encode_key(key), self.encode_value(value)))

    def hset(self, key, field, value):
        print('%s.%s = %s' % (self.encode_key(key), self.encode_key(field), self.encode_value(value)))

    def sadd(self, key, member):
        print('%s has {%s}' % (self.encode_key(key), self.encode_value(member)))

    def rpush(self, key, value):
        print('%s has [%s]' % (self.encode_key(key), self.encode_value(value)))

    def zadd(self, key, score, member):
        print('%s has {%s : %s}' % (str(key), str(member), str(score)))


callback = MyCallback()
parser = RdbParser(callback)
parser.parse('./dump.rdb')%     