Django shell_plus: How to access Jupyter notebook in Docker Container

Question

I am trying to access a Jupyter Notebook created with the shell_plus command from django-extensions in a Docker container.

docker-compose -f local.yml run --rm django python manage.py shell_plus --notebook

My configuration is based on the answers of @RobM and @Mark Chackerian to this Stack Overflow question. I.e. I installed and configured a custom kernel and my Django apps config file has the constant NOTEBOOK_ARGUMENTS set to:

NOTEBOOK_ARGUMENTS = [
    '--ip', '0.0.0.0',
    '--port', '8888',
    '--allow-root',
    '--no-browser',
]

I can see the container starting successfully in the logs:

[I 12:58:54.877 NotebookApp] The Jupyter Notebook is running at:
[I 12:58:54.877 NotebookApp] http://10d56bab37fc:8888/?token=b2678617ff4dcac7245d236b6302e57ba83a71cb6ea558c6
[I 12:58:54.877 NotebookApp]  or http://127.0.0.1:8888/?token=b2678617ff4dcac7245d236b6302e57ba83a71cb6ea558c6

But I can't open the url. I have forwarded the port 8888 in my docker-compose, tried to use localhost instead of 127.0.0.1 and also tried to use the containers IP w/o success.

It feels like I am missing the obvious here … Any help is appreciated.

Answer 1

For the sake of records as of 2020, I managed to have a working django setup with Postgresql in docker-compose:

development.py (settings.py)

INSTALLED_APPS += [
    "django_extensions",
]

SHELL_PLUS = "ipython"

SHELL_PLUS_PRINT_SQL = True

NOTEBOOK_ARGUMENTS = [
    "--ip",
    "0.0.0.0",
    "--port",
    "8888",
    "--allow-root",
    "--no-browser",
]

IPYTHON_ARGUMENTS = [
    "--ext",
    "django_extensions.management.notebook_extension",
    "--debug",
]

IPYTHON_KERNEL_DISPLAY_NAME = "Django Shell-Plus"

SHELL_PLUS_POST_IMPORTS = [ # extra things to import in notebook
    ("module1.submodule", ("func1", "func2", "class1", "etc")),
    ("module2.submodule", ("func1", "func2", "class1", "etc"))

]

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true" # only use in development 

requirements.txt

django-extensions
jupyter
notebook
Werkzeug  # needed for runserver_plus
...

docker-compose.yml

version: "3"

services:
  db:
    image: postgres:13
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
    restart: always
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data/
  web:
    build: .
    environment:
      - DJANGO_SETTINGS_MODULE=settings.development
    command:
      - scripts/startup.sh
    volumes:
      - ...
    ports:
      - "8000:8000" # webserver 
      - "8888:8888" # ipython notebook
    depends_on:
      - db

volumes:
  postgres_data:

From your host terminal run this command:

docker-compose exec web python manage.py shell_plus --notebook

Finally navigate to http://localhost:8888/?token=<xxxx> in the web browser of host.

Answer 2

Got it to work, but why it does so is beyond me. Exposing the ports in the docker-compose run command did the trick.

docker-compose -f local.yml run --rm -p 8888:8888 django python manage.py shell_plus --notebook

I was under the impression exposing ports in my local.yml would open them also in containers started by run.

Answer 3

If you want to use jupyter notebook like separated service:

jupyter_notebook:
  build:
    context: .
    dockerfile: docker/dev/web/Dockerfile
  command: python manage.py shell_plus --notebook
  depends_on:
    - web
  ports:
  - 8888:8888 # ipython notebook
  env_file:
    - .env

after:

docker-compose logs -f 'jupyter_notebook'

and you will get access token in logs

Answer 4

The compose run command will per default not expose the defined service ports. From the documentation at https://docs.docker.com/compose/reference/run/

The [...] difference is that the docker-compose run command does not create any of the ports specified in the service configuration. This prevents port collisions with already-open ports. If you do want the service’s ports to be created and mapped to the host, specify the --service-ports flag:

docker-compose run --service-ports web python manage.py shell 

You will therefore need to run

docker-compose -f local.yml run --rm --service-ports django python manage.py shell_plus --notebook

It might also be that the default 8888 port is already used by a local jupyter server (e.g. one spun up by VS Code's jupyter notebook implementation. I therefore usually map to a different port in the settings.py NOTEBOOK_ARGUMENTS list. (In this case the port mapping in the compose file needs to be adjusted as well, of course, and there must not be another container running in the background with the same service definition as this might also occupy the port.)