'Python redis-py multiprocessing: TypeError: cannot pickle '_thread.lock' object

I'm trying to parallelize a function in python which interacts with redis using redis-py package. To achieve this task I'm using multiprocessing pool, however I'm getting the TypeError: cannot pickle '_thread.lock' object error while doing it. The reason I'm getting this error is because, we cannot pickle StrictRedis() class instance. Kindly find my sample code for more information.

Note: Evidently, I searched on google and also explored the stackoverflow but no luck.

Sample Code:

from redis import StrictRedis
from multiprocessing import Pool


class Main:
    def __init__(self):
        self.r = StrictRedis("localhost", 6380)

    def func(self, x, y):
        z = self.r.get("z")
        print(x + y + z)

    def run(self):
        args = [("a", "b"), ("c", "d")]

        with Pool(10) as p:
            p.starmap(self.func, args)


m = Main()
m.run()

Solution 1: What I did is to initialize the StrictRedis() class in func() method like this in below snippet. But that'll create new instances of StrictRedis() for each processes and I don't find this adequate.

def func(self, x, y):
    r = StrictRedis("localhost", 6380)
    z = r.get("z")
    print(x + y + z)


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source