Category "parallel-processing"

SLURM nodes, tasks, cores, and cpus

Would someone be able to clarify what each of these things actually are? From what I gathered, nodes are computing points within the cluster, essentially a sing

How does OpenMP actually reduce clock cycles?

It might be a silly question but, with OpenMP you can achieve to distribute the number of operations between all the cores your CPU has. Of course, it is going

Batch Kronecker product of tensors

I have two tensors that are batches of matrices: x = torch.randn(100,10,10) y = torch.randn(100,2,2) I want to parallelize the kronecker on each matrix, not d

python parallel processing running all tasks on one core - multiprocessing, ray

I have a model.predict()-method and 65536 rows of data which takes about 7 seconds to perform. I wanted to speed this up using the joblib.parallel_backend tooli

How to avoid 'Too many open files' error when using parallelization within scipy.optimize.differential_evolution?

I am running a python script which uses scipy.optimize.differential_evolution to find optimum parameters for given data samples. I am processing my samples seq

How to run program in Bash script for as long as other program runs in parallel?

I have two programs server and client. server terminates after an unknown duration. I want to run client in parallel to server (both from the same Bash script)

Why stream created with Spliterators is not being processed in parallel?

This is probably very basic, but I am not a Java person. Here is my processing code which simply prints and sleeps: private static void myProcessings(int va

Python multiprocessing with macOs

I have a mac (MacOs 10.15.4, Python ver 3.82) and need to work in multiprocessing, but on my pc the procedures doesn’t work. For example, I have copied a

Running jest tests in parallel based on a list of parameters

In my case I have a test file containing a few hundred tests using jest describe('my test-suite', () => { test('test 1', () => { expect(1).toBe(

Keras model.fit() runs faster on GPU when the CPU is loaded with a heavy multiprocessing script

I wasn't expecting this to happen. The relevant code pieces are: import os import tensorflow as tf os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices' .

How to test the problem size scaling performance of code

I'm running a simple kernel which adds two streams of double-precision complex-values. I've parallelized it using OpenMP with custom scheduling: the slice_indic

FailSafe plugin is not running feature files in parallel

I have included failsafe plugin with parallel methods and threadcount 4. And framework is cucumber with junit. I'm trying to run features in parallel with fails

Server with multi-processor, how to launch code

I don't have a big knowledge in hardware, GPU and CPU so I'm trying to create it. I have a server with N processor, the description for each of them is more or

Why is my parallel std::for_each only using 1 thread?

I'm trying to parallelize this C++ code (computing a continuous Fourier transform of points, modeled as Dirac impulses), and this code compiles and works correc

GNU parallel - running command through multiple ssh jumphosts

I need to run commands on multiple hosts accessible via multiple jumphosts. I have a slf file ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker1

Should multithreading be used in microservices?

Should parallel programming be used in the development of microservices in case the microservices are scalable and, for instance, deployed as ECS on AWS? If ye

Parallelizing Monte Carlo Tree Search

I have a Monte Carlo Tree Search implementation that I need to optimize. So I thought about parallelizing the rollout phase. How to do that? (Is there a code ex

How to initialize workers to use package functions in parallel

I am developing an R package and trying to use parallel processing in it for an embarrassingly parallel problem. I would like to write a loop or functional that

How large should my list of objects be to warrant the use of java 8's parallelStream?

I have a list of objects from the database and i want to filter this list using the filter() method of the Stream class. New objects will be added to the databa

Unable to achieve async/parallel behavior inside for loop while using FastAPI and Ormar

I am using FastAPI and have an async route that needs to do many things like making calls to other API endpoints, and reading/writing to a database. It iterates