How to synchronize the output of Python subprocess

Question

I think I'm having issues to synchronize the output of two Popen running concurrently. It seems that the output from these two different command lines are interleaved with one another. I also tried using RLock to prevent this from happening but it didn't work.

A sample output would be:

cmd1
cmd1
cmd2
cmd2
cmd2
cmd2
cmd1
cmd2

The code is as attached:

import subprocess
import threading

class PopenWorkerThread(threading.Thread):
    def __init__(self, cmdLine):
        self.lock = threading.RLock()
        self.WebSphereCmdLine = cmdLine
        threading.Thread.__init__(self)

    def run(self):
        logger.error('Runninf: ' + self.WebSphereCmdLine)
        proc = subprocess.Popen(self.WebSphereCmdLine, shell=False, stdout=subprocess.PIPE)
        while True:
            self.lock.acquire()
            print proc.stdout.readline()
            self.lock.release()

def launch():
    commandLine = ['ls -l', 'netstat']
    for cmdLine in commandLine:
        workerThread = PopenWorkerThread(cmdLine)
        workerThread.start()

launch()

Is there a way to synchronize the outputs so that they look like such?

cmd1
cmd1
cmd1
cmd1
cmd1
cmd2
cmd2
cmd2
cmd2
cmd2

Answer 1

You're locking with a granularity of a line, so of course lines from one thread can and do alternate with lines from the other. As long as you're willing to wait until a process ends before showing any of its output, you can lock with the larger "process" granularity. Of course you have to use the SAME lock for both threads -- having each thread use a completely separate lock, as you're doing now, cannot achieve anything at all, obviously.

So, for example:

import subprocess
import threading

class PopenWorkerThread(threading.Thread):

    lock = threading.RLock()  # per-class, NOT per-instance!

    def __init__(self, cmdLine):
        self.WebSphereCmdLine = cmdLine
        threading.Thread.__init__(self)

    def run(self):
        logger.error('Runninf: ' + self.WebSphereCmdLine)
        proc = subprocess.Popen(self.WebSphereCmdLine, shell=False, stdout=subprocess.PIPE)
        result, _ = proc.communicate()
        with self.lock:
            print result,

def launch():
    commandLine = ['ls -l', 'netstat']
    for cmdLine in commandLine:
        workerThread = PopenWorkerThread(cmdLine)
        workerThread.start()

launch()

Answer 2

Maybe you are looking for the wait method

http://docs.python.org/library/subprocess.html#subprocess.Popen.wait