Jupyter installation: [Errno 99] Cannot assign requested address



  • Avatar
    Gianfranco Campana

    Here the containers full command exposing the ports:

    CONTAINER ID                                                       IMAGE                                 COMMAND                                                                                                                         CREATED          STATUS                      PORTS                                                                                                      NAMES
    3a7ea963f8256d97fdf563223426af70b382d7bef9806d9dccf813d97cdb5128   omnisci/omnisci-jupyterhub:v0.4       "/docker-entrypoint.sh jupyterhub -f /srv/jupyterhub/jupyterhub_config.py"                                                      10 minutes ago   Up 10 minutes               8000/tcp                                                                                                   jupyterhub
    d1a1671cff2f96322bcdd00030f3d2abc129fa9097ac27bb922148e2819483c0   omnisci/omnisci-jupyterlab-cpu:v0.5   "tini -g -- echo"                                                                                                               10 minutes ago   Exited (0) 10 minutes ago                                                                                                              heavyai_jupyterlab-tmp_1
    da84e0f600e13f23bc63c1f3752c5b3621115bb51a3dcf5207d835f7ac5a345f   heavyai/heavyai-ee-cuda:latest        "/bin/sh -c '/opt/heavyai/startheavy --non-interactive --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf'"   10 minutes ago   Up 10 minutes     >6273-6274/tcp, :::6273-6274->6273-6274/tcp,>6278/tcp, :::6278->6278/tcp   heavyaiserver

    Here the jupyrterhub container log:

    docker logs --tail 250 --follow --timestamps 3a7ea963f825
    2022-09-08T15:40:57.275859002Z Generating a RSA private key
    2022-09-08T15:40:57.952654793Z .................................................................................................................................................++++
    2022-09-08T15:40:57.969533364Z ..++++
    2022-09-08T15:40:57.970084383Z writing new private key to '/etc/ssl/private/jhubcert.key'
    2022-09-08T15:40:57.970499210Z -----
    2022-09-08T15:40:59.357718145Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Authenticator: jhub-omnisci-authenticator.omnisci_user_auth.OmniSciUserAuthenticator
    2022-09-08T15:40:59.357749367Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
    2022-09-08T15:40:59.362819375Z [I 2022-09-08 15:40:59.362 JupyterHub app:1302] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
    2022-09-08T15:40:59.386177869Z [I 2022-09-08 15:40:59.385 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
    2022-09-08T15:40:59.386205151Z [I 2022-09-08 15:40:59.386 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
    2022-09-08T15:40:59.421206486Z [I 2022-09-08 15:40:59.420 alembic.runtime.migration migration:356] Running stamp_revision  -> 4dc2d5a8c53c
    2022-09-08T15:40:59.789274724Z [I 2022-09-08 15:40:59.788 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
    2022-09-08T15:40:59.817261218Z [I 2022-09-08 15:40:59.817 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
    2022-09-08T15:40:59.943050210Z [I 2022-09-08 15:40:59.942 JupyterHub app:2337] Hub API listening on
    2022-09-08T15:40:59.944336999Z [W 2022-09-08 15:40:59.944 JupyterHub proxy:642] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
    2022-09-08T15:40:59.944361212Z [I 2022-09-08 15:40:59.944 JupyterHub proxy:645] Starting proxy @ http://:8000/jupyter/
    2022-09-08T15:41:00.554298654Z 15:41:00.552 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
    2022-09-08T15:41:00.554783154Z 15:41:00.554 - info: [ConfigProxy] Proxy API at
    2022-09-08T15:41:00.583033824Z [I 2022-09-08 15:41:00.582 JupyterHub app:2362] Starting managed service cull_idle
    2022-09-08T15:41:00.583420371Z [I 2022-09-08 15:41:00.583 JupyterHub service:316] Starting service 'cull_idle': ['python', '/srv/jupyterhub/cull_idle_servers.py', '--timeout=3600']
    2022-09-08T15:41:00.590010542Z [I 2022-09-08 15:41:00.589 JupyterHub service:121] Spawning python /srv/jupyterhub/cull_idle_servers.py --timeout=3600
    2022-09-08T15:41:00.606208627Z [I 2022-09-08 15:41:00.605 JupyterHub proxy:319] Checking routes
    2022-09-08T15:41:00.606238386Z [I 2022-09-08 15:41:00.606 JupyterHub proxy:399] Adding default route for Hub: /jupyter/ =>
    2022-09-08T15:41:00.609329584Z [I 2022-09-08 15:41:00.609 JupyterHub app:2422] JupyterHub is now running at http://:8000/jupyter/
    2022-09-08T15:41:00.808231000Z [I 2022-09-08 15:41:00.807 JupyterHub log:174] 200 GET /jupyter/hub/api/users (cull_idle@ 40.23ms
    2022-09-08T15:42:18.743364593Z [E 2022-09-08 15:42:18.742 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address
    2022-09-08T15:42:18.743437615Z [W 2022-09-08 15:42:18.743 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true ( Error with OmniSci authentication: [Errno 99] Cannot assign requested address
    2022-09-08T15:42:18.794422406Z [W 2022-09-08 15:42:18.794 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@ 54.86ms
    2022-09-08T15:42:33.943313935Z [E 2022-09-08 15:42:33.942 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address
    2022-09-08T15:42:33.943363026Z [W 2022-09-08 15:42:33.943 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true ( Error with OmniSci authentication: [Errno 99] Cannot assign requested address
    2022-09-08T15:42:33.946198452Z [W 2022-09-08 15:42:33.945 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@ 5.52ms
    Comment actions Permalink
  • Avatar
    Gianfranco Campana

    Any help? Anyone got the same problem installing Jupyter for Immerse?
    This is still happening also after reinstalled Ubuntu, same identical error, and therefore the problem is easily reproducible with the configuration I'm using.

    In addition there is an open point about " overriding the default start command " since when this problem appeared :

    There is a best practice I missed? Maybe with a different configuration the problem would not appear?

    Comment actions Permalink
  • Avatar
    Gianfranco Campana

    After a ton of testing and debugging i got Docker/Jupyter/heavy db working: I used a number of configuration parameters that are different from what is documented in online docs.

    Starting from the revision made in docker-compose.yml revised shared by Neill in this post , I also modified in docker-compose.yml:

    • heavyaiserver: container_name: heavyaiserver --> omnisciserver
    • HEAVYAI_HOST: heavyaiserver --> OMNISCI_HOST: heavyaiserver

    • \#JLAB_DOCKER_NETWORK_NAME: heavyai-backend --> uncomment this

    With these parameters in place the notebook is spawned correctly: image|677x500, 100%

    I thought it might be useful for other users out there ;)

    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Hi @Gianfranco_Campana,

    I suggest you leave the container names unchanged and do an override on defaults using this yaml file

    version: '3.7'
        container_name: heavyaiserver
        image: heavyai/heavyai-ee-cuda:v6.0.0
        restart: always
        ipc: shareable
          - /var/lib/heavyai:/var/lib/heavyai
          - /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox:/jhub_heavyai_dropbox
          - heavyai-frontend
          - heavyai-backend
          - "6273:6273"
          - "6274:6274"
          - "6278:6278"
        # If using binary encryption, uncomment the below line to override the default 
        # command that uses startheavyai, noting that you must have an existing HEAVY.AI 
        # data directory and run initdb before making this change.
        # command: /bin/bash -c "/heavyai/bin/heavy_web_server --config /heavyai-storage/heavyai.conf & /heavyai/bin/heavydb --config /heavyai-storage/heavyai.conf"
      # The purpose of this is to make sure the jlabimage is pulled because jupyterhub will not pull it automatically when launching
        image: &jlabimage omnisci/omnisci-jupyterlab-cpu:v0.5
        command: echo
            - heavyai-backend
        container_name: jupyterhub
        image: omnisci/omnisci-jupyterhub:v0.4
        restart: always
          - heavyai-backend
          - heavyaiserver
          - /var/run/docker.sock:/var/run/docker.sock
          # Map this volume if binary encryption mode is configured and certificates are being validated
          # - /var/lib/heavyai/cacerts.crt:/heavyai-storage/cacerts.crt
          ### Required settings ###
          OMNISCIDB_CONTAINER_NAME: heavyaiserver
          SPAWNER_ENV_OMNISCI_HOST: heavyaiserver
          DOCKER_JUPYTER_LAB_IMAGE: *jlabimage
          OMNISCI_HOST: heavyaiserver
          OMNISCI_JUPYTER_ROLE: heavyai_jupyter
          JLAB_DOCKER_NETWORK_NAME: heavyai-backend
          ### Optional settings  ###
          # DOCKER_NOTEBOOK_DIR: /home/jovyan                   # The directory inside the user's Jupyter Lab container to mount the user volume to.
          # HUB_IP: jupyterhub                                  # The hostname or IP of the Jupyter Hub server
          # JHUB_BASE_URL: /jupyter/                            # The base URL prepended to all Jupyter Hub and Lab requests
          # HEAVYAI_ALLOW_SUPERUSER_ROLE: "false"               # Enable / disable admin access to Jupyter Hub
          # JLAB_CONTAINER_AUTOREMOVE: "true"                   # Enable / disable automatic removal of stopped Juptyer Lab containers
          # JLAB_DOCKER_NETWORK_NAME: heavyai-backend           # The docker network name for Jupyter Lab containers
          # JLAB_IDLE_TIMEOUT: 3600                             # Shut down Jupyter Lab containers after this many seconds of idle time
          # JLAB_NAME_PREFIX: jupyterlab                        # Container name prefix for Lab containers
          # JLAB_HEAVYAI_IMPORT_VOLUME_PATH: /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox  # Local Docker host path for where to mount the shared directory available to the HeavyDB server for file ingest
          # JLAB_NOTEBOOK_TERMINALS_ENBLED: "false"             # Enable terminals in notebooks
          # JLAB_USER_VOLUME_PATH: /var/lib/heavyai/jupyter/    # Local Docker host path to be used for user Jupyter Lab home directory volume mapping
          # JUPYTER_DEBUG: "true"                               # Turn on / off debugging for Jupyter Hub and Lab
          # OMNISCI_BINARY_TLS_CACERTS_LOCAL_PATH: /var/lib/heavyai/cacerts.crt # Specifying this or mapping a volume in the Hub container to /heavyai-storage/cacerts.crt will automatically enable binary TLS mode
          # OMNISCI_BINARY_TLS_VALIDATE: "false" # Whether or not to validate certificates in binary TLS mode. Specifying either "true" or "false" will enable binary TLS mode
          # OMNISCI_PORT: 6278 # Port that Jupyter Hub should use to connect to HEAVY.AI. Ensure this matches the protocol
          # OMNISCI_PROTOCOL: http # Protocol that Jupyter Hub should use to connect to HEAVY.AI. Ensure this is "binary" if using binary encryption
          # SPAWNER_CPU_LIMIT: 1                                # Number of CPU cores available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.cpu_limit
          # SPAWNER_RAM_LIMIT: 10G                              # Amount of CPU RAM available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.LocalProcessSpawner.mem_limit
          # SPAWNER_ENV_OMNISCI_HOST: heavyaiserver             # Hostname / IP address of the HEAVY.AI server for Lab containers to connect to by default
          # SPAWNER_ENV_OMNISCI_PORT: 6274                      # Port of the HEAVY.AI server for Lab containers to connect to by default
          # SPAWNER_ENV_OMNISCI_PROTOCOL: binary                # Protocol of the HEAVY.AI server for Lab containers to connect to by default
          # OMNISCI_DB_URL: "heavyai://heavyaiserver:6274/heavyai" # Alternative, direct connection (not Immerse session) to heavyaiserver. Username will be inferred, but password will be required in notebook.
          # More volumes for lab containers:
          # JLAB_VOLUME_1: /data1:/data1:rw
          # JLAB_VOLUME_2: /var/lib/heavyai:/heavyai-storage:ro
        driver: bridge
        name: heavyai-frontend
        driver: bridge
        name: heavyai-backend

    we are modìifying the docs as a first step this way.

    Many thanks for you precious contribution


    Comment actions Permalink
  • Avatar
    Gianfranco Campana

    Hi Candido, just to let you know that with

    OMNISCI_JUPYTER_ROLE: heavyai_jupyter ```
    assigning heavyai_jupyter role to the user, the Jupyter button disappears, while with :

    OMNISCI_JUPYTER_ROLE: omnisci_jupyter```

    assigning omnisci_jupyter role to the user, it is shown.

    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Hi Gianfranco,

    I'll check it out because I think I have used the heavyai_jupyter role for the user. I'll back to you when checked, in the meantime thanks for the help.


    Comment actions Permalink

Please sign in to leave a comment.