'Python redis-py multiprocessing: TypeError: cannot pickle '_thread.lock' object
I'm trying to parallelize a function in python which interacts with redis using redis-py package. To achieve this task I'm using multiprocessing pool, however I'm getting the TypeError: cannot pickle '_thread.lock' object
error while doing it. The reason I'm getting this error is because, we cannot pickle StrictRedis()
class instance. Kindly find my sample code for more information.
Note: Evidently, I searched on google and also explored the stackoverflow but no luck.
Sample Code:
from redis import StrictRedis
from multiprocessing import Pool
class Main:
def __init__(self):
self.r = StrictRedis("localhost", 6380)
def func(self, x, y):
z = self.r.get("z")
print(x + y + z)
def run(self):
args = [("a", "b"), ("c", "d")]
with Pool(10) as p:
p.starmap(self.func, args)
m = Main()
m.run()
Solution 1:
What I did is to initialize the StrictRedis()
class in func() method like this in below snippet. But that'll create new instances of StrictRedis()
for each processes and I don't find this adequate.
def func(self, x, y):
r = StrictRedis("localhost", 6380)
z = r.get("z")
print(x + y + z)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|