Connecting Spyder to Remote Jupyter Notebook in a Docker Container

Question

I have been trying to connect Spyder to a docker container running on a remote server and failing time and again. Here is a quick diagram of what I am trying to achieve:

Currently I am launching the docker container on the remote machine through ssh with

docker run --runtime=nvidia -it --rm --shm-size=2g -v /home/timo/storage:/storage -v /etc/passwd:/etc/passwd -v /etc/group:/etc/group --ulimit memlock=-1 -p 8888:8888 --ipc=host ufoym/deepo:all-jupyter

so I am forwarding on port 8888. Then inside the docker container I am running

jupyter notebook --no-browser --ip=0.0.0.0 --port=8888 --allow-root --notebook-dir='/storage' 

OK, now for the Spyder part - As per the instructions here, I go to ~/.local/share/jupyter/runtime, where I find the following files:

kernel-ada17ae4-e8c3-4e17-9f8f-1c029c56b4f0.json  nbserver-11-open.html  nbserver-21-open.html  notebook_cookie_secret
kernel-e81bc397-05b5-4710-89b6-2aa2adab5f9c.json  nbserver-11.json       nbserver-21.json

Not knowing which one to take, I copy them all to my local machine.

I now go to Consoles->Connect to an Existing Kernel, which gives me the "Connect to an Existing Kernel" window which I fill out as so (of course using my actual remote IP address):

(here I have chosen the first of the json files for Connection info:). I hit enter and Spyder goes dark and crashes.

This happens regardless of which connection info file I choose. So, my questions are:

1: Am I doing all of this correctly? I have found lots of instructions for how to connect to remote servers, but not so far for specifically connecting to a jupyter notebook on a docker on a remote server.

2: If yes, then what else can I do to troubleshoot the issues I am encountering?

I should also note that I have no problems connecting to the Jupyter Notebook through the browser on my local machine. It's just that I would prefer to be working with Spyder as my IDE.

Many thanks in advance!

Answer 1

This isn't a solution so much as a work around, but sshfs might be of help

Use sshfs to mount the remote machine's home directory on a local directory, then your local copy of Spyder can edit the file as if it were a local file.

sshfs remotehost.com:/home/user/ ./remote-host/

It typically takes about half a second to upload the changes to an AWS host when you I hit save in Spyder, which is an acceptable delay for me. When it's time to run the code, ssh into the remote machine, and run the code from an IPython shell. It's not elegant, but it does work.

I'm not expecting this to be the best answer, but maybe you can use it as a stopgap solution.

Answer 2

I have the same problem with you. I got it working, maybe a bit clumsy as I am totally new to docker. Here are my steps and notes on where we differ, hope this helps:

• Launch docker conatiner in remote machine:

  docker run --gpus all --rm -ti --net=host -v /my_storage/data:/home/data -v /my_storage/JSON:/root/.local/share/jupyter/runtime repo/tensorflow:20.03-tf2-py3

I use a second volume mount, in order to get kernel.json file to my local computer. I couldn't manage to access directly from docker via ssh, as it is in /root/ folder in docker container, and with root-only access. If you know how to read from there directly, I'll be happy to learn. My workaround is: On remote machine, create a JSON/ directory, and map it to the "jupyter --runtime-dir" in container. Once the kernel is created, access the kernel-xxx.json file through this volume mount, copy to local machine and chmod.

• Launch ipython kernel in container: ipython kernel You are launching jupyter notebook. I suspect this is the reason for your problem. I am not sure if spyder works on notebooks, but it works on iPython kernels. Probably, it works better on spyder-kernels.

• copy kernel.json file from /remote_machine/JSON to local machine, chmod for accessing.

• launch spyder, use local kernel.json and ssh settings. This part is same as yours.

Answer 3

Not enough reputation... to add comment but to chime on @asim's solution. I was able to have my locally installed Spyder to connect to a kernel running from a container on a remote machine. There is bit of manual work but I am okay with this since I can get much more done with Spyder than with other IDEs.

docker run --rm -it --net=host -v /project_directory_remote_machine:/container_project_directory image_id bash

from container

python -m spyder_kernels.console - matplotlib=’inline’ --ip=127.0.0.1 -f=/container_project_directory/connection_file.json

from remote machine, chmod connection_file.json to open then open and copy/paste content to a file on a local machine :) Use the json file to connect to a remote kernel following steps in the sources below

https://medium.com/@halmubarak/connecting-spyder-ide-to-a-remote-ipython-kernel-25a322f2b2be https://mazzine.medium.com/how-to-connect-your-spyder-ide-to-an-external-ipython-kernel-with-ssh-putty-tunnel-e1c679e44154