Category "python-multiprocessing"

Python multiprocessing, parent process hangs at recv after child process raises exception before send

Python is definitely one of the worst languages for parallel processing. Just finna get the parent process to throw an error when the child process fails. Inste

Python redis-py multiprocessing: TypeError: cannot pickle '_thread.lock' object

I'm trying to parallelize a function in python which interacts with redis using redis-py package. To achieve this task I'm using multiprocessing pool, however I

python parallel processing running all tasks on one core - multiprocessing, ray

I have a model.predict()-method and 65536 rows of data which takes about 7 seconds to perform. I wanted to speed this up using the joblib.parallel_backend tooli

How to avoid excessive ram consumption using pathos

This is a rough example of how I leverage multiprocessing with pathos: from pathos.multiprocessing import ProcessingPool pool = ProcessingPool(10) results = po

Exception ignored in: <function Pool.__del__ at 0x7fbf30520550>

I'm using TensorFlow training a deep learning model and the model is successfully trained however at the end it returns this error message to me: Exception igno

Python multiprocessing with TensorRT

I am trying to use a TensorRT engine for inference in a python class that inherits from multiprocessing. The engine works in a standalone python script on my sy

scipy.fftpack.fft with multiprocessing, how to avoid performance losses?

I would like to use scipy.fftpack.fft (and rfft) inside a multiprocessing structure, I have observed significant performances losses due to an apparent incompat

Python2: multiprocessing.dummy.Pool vs multiprocessing.pool.ThreadPool

In python 2, is there any difference between multiprocessing.dummy.Pool and multiprocessing.pool.ThreadPool? The source code seems to imply they're the same.

Spacy train ner using multiprocessing

I am trying to train a custom ner model using spacy. Currently, I have more than 2k records for training and each text consists of more than 100 words, at least

Read files with ZipFile using multiprocessing

I'm trying to read raw data from a zipfile. The structure of that file is: zipfile data Spectral0.data Spectral1.data Spectral[...].data Spectral300.data Heade

Python 3.6+: Nested multiprocessing managers cause FileNotFoundError

So I'm trying to use multiprocessing Manager on a dict of dicts, this was my initial try: from multiprocessing import Process, Manager def task(stat): tes

Best way to wait for queue population python multiprocessing

It is first time I play with parallel computing seriously. I am using multiprocessing module in python and I am running into this problem: A queue consumer run

Python multiprocessing shared memory error on close

I am using Python multiprocessing shared memory. I see the following error when I try to close a SharedMemory object: BufferError: memoryview has 1 exported buf

Python Multiprocessing concurrency using Manager, Pool and a shared list not working

I am learning python multiprocessing, and I am trying to use this feature to populate a list with all the files present in an os. However, the code that I wrote

How to use multiprocessing.Event in an asyncio event loop?

I am using the multiprocessing module to create a Process. In this child process, it will run an asyncio event loop, and in the parent process, I am not. How ca

Parallelize Python Dictionary Comprehension

I'm trying to parallelize the subsetting of a Python dictionary. The code below creates a new dictionary, positions_sub, based on if the keys in positions dicti

Other Categories