Reading command output with Paramiko invoke_shell/send/recv never finishes

Question

I am trying to use send/recv function in Paramiko. According to what I see, the line throws an exception of timeout

Evaluating: self.shell.recv(1024) did not finish after 3.00 seconds.

tmp = shell.recv(1024)

What is wrong with the function implementation? My exit condition from while True is an exception, how can I change that to exit without an exception?

Full code:

self.shell = self.SSHConnect(ip, username, password)

def SSHConnect(self, ip, username, passowrd):
    ssh = paramiko.SSHClient()
    LOGGER.debug(msg="Open SSH Client to :" + str(ip))
    try:
        ssh.set_missing_host_key_policy(policy=paramiko.AutoAddPolicy())
        ssh.connect(ip, port=22, username=username, password=passowrd, allow_agent=False, look_for_keys=True)
        if self.device_type != 'linux_host':
            session = ssh.invoke_shell()
            return session
    except Exception as ex:
        LOGGER.critical(msg="SSH Client wasn't established! Device name : " + str(self.device_name))
        return None 
    
    LOGGER.info(msg="Open SSH Client to :" + str(ip) + " established!")
    return ssh

def run_command(self,cmd):
    # New run command without throwing exception at the end:
    LOGGER.debug('Start new Run command with cmd =  ' + str(cmd))
    try:
        #Check the shell is activated before sending command:
        LOGGER.debug('Check the shell is activated before sending command: ' + cmd)
        if self.shell.get_transport().active:
            LOGGER.debug('Shell is Activated ! Running the command ')
            if self.device_type == 'linux_host':
                stdin, stdout, stderr = self.shell.exec_command(cmd)
            else:
                try:
                    #Command for switch of UFMAPL
                    LOGGER.debug('Sending command to UFMAPL/Switch with send()')
                    out = ''
                    self.shell.send(cmd)
                    while not self.shell.recv_ready():
                        time.sleep(3)
                    counter = 1
                    print('\ncommand is : ' + cmd + '\n' )
                    while True:
                        try:
                            print('iteration number is : #' + str(counter))
                            tmp = self.shell.recv(1024)
                            counter = counter + 1 
                            if not tmp:
                                break
                        except Exception as e:
                            break
                        out += tmp.decode("utf-8")
                        print('After iteration #' + str(counter) + ' out = ' + out + '\n\n')
                    ansi_escape = re.compile(r'\x1B\[[0-?]*[ -/]*[@-~]')
                    out = ansi_escape.sub('', out)
                    print('Printing final value before return : ' + str(out +'\n'))
                    return out
                except Exception as e:
                    LOGGER.error('Exception in send() : ' +str(e) )
                    return None
        else:
            LOGGER.critical('Shell is not activated !')
            return ""
    
        if stderr.read():
            LOGGER.critical('stderr is not empty which means the last command was failed, the command might not exist on host/switch ' )
            return stderr.read().decode('utf-8')
    
        out = stdout.read()
        if out:
            return out.decode("utf-8")
        else:
            LOGGER.critical('Run command sucussfully but return empty...')
            return out.decode("utf-8")
    except Exception as e:
        LOGGER.error('Exception received in run command : ' + str(e))

Logs (Printing to screen):

IM HEREEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE


command is : enable


iteration number is : #1

After iteration #2 out =
UFM Appliance

UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] >


iteration number is : #2

After iteration #3 out =
UFM Appliance

UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > e


iteration number is : #3

After iteration #4 out =
UFM Appliance

UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-a


iteration number is : #4

After iteration #5 out = 
UFM Appliance

UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl00


iteration number is : #5

After iteration #6 out =
UFM Appliance

UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl008-gen2 [ mgmt-sa ] #


iteration number is : #6

As you can see the debugger is stuck on iteration #6 (freeze). Why does it freeze and doesn't send output?

Enviroment details:

• Windows 10
• Eclipse latest

I would appreciate any help here. Let me know if you need more details.

Answer 1

Your code gets to the point, where the server stops waiting for another input. At the same time you wait for the server to output something. What it never will. That's called a deadlock.

You might have expected some kind of signal that the first command execution has finished. There's no such signal. You are using a "shell" (SSHClient.invoke_shell). The shell is a black box with an input and an output. There are no other signals except for the output, wihhc you are reading already.

The "shell" should not be used to automate command execution. For command automation, there's the "exec" channel (SSHClient.exec_command in Paramiko).

Though I understand that with some special devices, what your server seems to be, you might not have any other option (see Executing command using Paramiko exec_command on device is not working). Moreover I'm not ever sure how the enable command works. Whether it's a command that has finished, or whether it started a kind of a new shell, which is still running and is waiting for subcommands.

So in the end all you can do is to parse the output, waiting for the command prompt (smg-ib-apl008-gen2 [ mgmt-sa ] #).

---

Yes, it's ugly. You are trying to automate something that was not intended for automation. Maybe your server has a better API then enable shell command, that would be nicer to automate. But that's for another question. From SSH/Python/Paramiko point of view, there's no better solution, if you need to stick with executing the enable command in the shell.

---

Related questions:

• How to get each dependent command execution output using Paramiko exec_command
• Execute multiple dependent commands individually with Paramiko and find out when each command finishes

Though they are about more regular servers, like Linux. So they won't really help. I'm linking them just to provide broader picture and context.