Jupyter installation: [Errno 99] Cannot assign requested address

September 06, 2022 09:20

I just installed and configured Docker/Jupyter/Heavyai on Ubuntu 20.04.

Heavydb and Immerse are working fine, and I can see the Jupyter button in Immerse.

But opening Jupyter return this error:

401 : Unauthorized
Error with OmniSci authentication: [Errno 99] Cannot assign requested address

I can see the same error also in the container log:

2022-09-05T08:36:52.345705462Z 2022-09-05T08:36:52.345389 E 17 0 7 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278
2022-09-05T08:36:52.345742561Z 2022-09-05T08:36:52.345530 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274
2022-09-05T08:36:52.345926321Z 2022-09-05T08:36:52.345464 E 17 0 8 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276

here the configuration file currently used: docker-compose.yml.txt|attachment (5.2 KB) heavyai.conf.txt|attachment (119 Bytes) servers.json.txt|attachment (38 Bytes)

Comments

6 comments

Gianfranco Campana

September 08, 2022 15:57

Here the containers full command exposing the ports:

CONTAINER ID                                                       IMAGE                                 COMMAND                                                                                                                         CREATED          STATUS                      PORTS                                                                                                      NAMES
3a7ea963f8256d97fdf563223426af70b382d7bef9806d9dccf813d97cdb5128   omnisci/omnisci-jupyterhub:v0.4       "/docker-entrypoint.sh jupyterhub -f /srv/jupyterhub/jupyterhub_config.py"                                                      10 minutes ago   Up 10 minutes               8000/tcp                                                                                                   jupyterhub
d1a1671cff2f96322bcdd00030f3d2abc129fa9097ac27bb922148e2819483c0   omnisci/omnisci-jupyterlab-cpu:v0.5   "tini -g -- echo"                                                                                                               10 minutes ago   Exited (0) 10 minutes ago                                                                                                              heavyai_jupyterlab-tmp_1
da84e0f600e13f23bc63c1f3752c5b3621115bb51a3dcf5207d835f7ac5a345f   heavyai/heavyai-ee-cuda:latest        "/bin/sh -c '/opt/heavyai/startheavy --non-interactive --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf'"   10 minutes ago   Up 10 minutes               0.0.0.0:6273-6274->6273-6274/tcp, :::6273-6274->6273-6274/tcp, 0.0.0.0:6278->6278/tcp, :::6278->6278/tcp   heavyaiserver

Here the jupyrterhub container log:

docker logs --tail 250 --follow --timestamps 3a7ea963f825
2022-09-08T15:40:57.275859002Z Generating a RSA private key
2022-09-08T15:40:57.952654793Z .................................................................................................................................................++++
2022-09-08T15:40:57.969533364Z ..++++
2022-09-08T15:40:57.970084383Z writing new private key to '/etc/ssl/private/jhubcert.key'
2022-09-08T15:40:57.970499210Z -----
2022-09-08T15:40:59.357718145Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Authenticator: jhub-omnisci-authenticator.omnisci_user_auth.OmniSciUserAuthenticator
2022-09-08T15:40:59.357749367Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
2022-09-08T15:40:59.362819375Z [I 2022-09-08 15:40:59.362 JupyterHub app:1302] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
2022-09-08T15:40:59.386177869Z [I 2022-09-08 15:40:59.385 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
2022-09-08T15:40:59.386205151Z [I 2022-09-08 15:40:59.386 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
2022-09-08T15:40:59.421206486Z [I 2022-09-08 15:40:59.420 alembic.runtime.migration migration:356] Running stamp_revision  -> 4dc2d5a8c53c
2022-09-08T15:40:59.789274724Z [I 2022-09-08 15:40:59.788 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
2022-09-08T15:40:59.817261218Z [I 2022-09-08 15:40:59.817 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
2022-09-08T15:40:59.943050210Z [I 2022-09-08 15:40:59.942 JupyterHub app:2337] Hub API listening on http://172.19.0.3:8081/jupyter/hub/
2022-09-08T15:40:59.944336999Z [W 2022-09-08 15:40:59.944 JupyterHub proxy:642] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
2022-09-08T15:40:59.944361212Z [I 2022-09-08 15:40:59.944 JupyterHub proxy:645] Starting proxy @ http://:8000/jupyter/
2022-09-08T15:41:00.554298654Z 15:41:00.552 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
2022-09-08T15:41:00.554783154Z 15:41:00.554 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
2022-09-08T15:41:00.583033824Z [I 2022-09-08 15:41:00.582 JupyterHub app:2362] Starting managed service cull_idle
2022-09-08T15:41:00.583420371Z [I 2022-09-08 15:41:00.583 JupyterHub service:316] Starting service 'cull_idle': ['python', '/srv/jupyterhub/cull_idle_servers.py', '--timeout=3600']
2022-09-08T15:41:00.590010542Z [I 2022-09-08 15:41:00.589 JupyterHub service:121] Spawning python /srv/jupyterhub/cull_idle_servers.py --timeout=3600
2022-09-08T15:41:00.606208627Z [I 2022-09-08 15:41:00.605 JupyterHub proxy:319] Checking routes
2022-09-08T15:41:00.606238386Z [I 2022-09-08 15:41:00.606 JupyterHub proxy:399] Adding default route for Hub: /jupyter/ => http://172.19.0.3:8081
2022-09-08T15:41:00.609329584Z [I 2022-09-08 15:41:00.609 JupyterHub app:2422] JupyterHub is now running at http://:8000/jupyter/
2022-09-08T15:41:00.808231000Z [I 2022-09-08 15:41:00.807 JupyterHub log:174] 200 GET /jupyter/hub/api/users (cull_idle@172.19.0.3) 40.23ms
2022-09-08T15:42:18.743364593Z [E 2022-09-08 15:42:18.742 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address
2022-09-08T15:42:18.743437615Z [W 2022-09-08 15:42:18.743 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true (172.19.0.1): Error with OmniSci authentication: [Errno 99] Cannot assign requested address
2022-09-08T15:42:18.794422406Z [W 2022-09-08 15:42:18.794 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@172.19.0.1) 54.86ms
2022-09-08T15:42:33.943313935Z [E 2022-09-08 15:42:33.942 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address
2022-09-08T15:42:33.943363026Z [W 2022-09-08 15:42:33.943 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true (172.19.0.1): Error with OmniSci authentication: [Errno 99] Cannot assign requested address
2022-09-08T15:42:33.946198452Z [W 2022-09-08 15:42:33.945 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@172.19.0.1) 5.52ms

Gianfranco Campana

September 27, 2022 16:17
Any help? Anyone got the same problem installing Jupyter for Immerse?
This is still happening also after reinstalled Ubuntu, same identical error, and therefore the problem is easily reproducible with the configuration I'm using.

In addition there is an open point about " overriding the default start command " since when this problem appeared :

There is a best practice I missed? Maybe with a different configuration the problem would not appear?
0

Comment actions Permalink
Gianfranco Campana

September 29, 2022 15:37
After a ton of testing and debugging i got Docker/Jupyter/heavy db working: I used a number of configuration parameters that are different from what is documented in online docs.

Starting from the revision made in docker-compose.yml revised shared by Neill in this post , I also modified in docker-compose.yml:
- heavyaiserver: container_name: heavyaiserver --> omnisciserver
- HEAVYAI_HOST: heavyaiserver --> OMNISCI_HOST: heavyaiserver
- \#JLAB_DOCKER_NETWORK_NAME: heavyai-backend --> uncomment this
With these parameters in place the notebook is spawned correctly:

I thought it might be useful for other users out there ;)
1

Comment actions Permalink

Candido Dessanti

September 30, 2022 19:45

Hi @Gianfranco_Campana,

I suggest you leave the container names unchanged and do an override on defaults using this yaml file

version: '3.7'

services:

  heavyaiserver:
    container_name: heavyaiserver
    image: heavyai/heavyai-ee-cuda:v6.0.0
    restart: always
    ipc: shareable
    volumes:
      - /var/lib/heavyai:/var/lib/heavyai
      - /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox:/jhub_heavyai_dropbox
    networks:
      - heavyai-frontend
      - heavyai-backend
    ports:
      - "6273:6273"
      - "6274:6274"
      - "6278:6278"
    # If using binary encryption, uncomment the below line to override the default 
    # command that uses startheavyai, noting that you must have an existing HEAVY.AI 
    # data directory and run initdb before making this change.
    # command: /bin/bash -c "/heavyai/bin/heavy_web_server --config /heavyai-storage/heavyai.conf & /heavyai/bin/heavydb --config /heavyai-storage/heavyai.conf"


  # The purpose of this is to make sure the jlabimage is pulled because jupyterhub will not pull it automatically when launching
  jupyterlab-tmp:
    image: &jlabimage omnisci/omnisci-jupyterlab-cpu:v0.5
    command: echo
    networks:
        - heavyai-backend


  jupyterhub:
    container_name: jupyterhub
    image: omnisci/omnisci-jupyterhub:v0.4
    restart: always
    networks:
      - heavyai-backend
    depends_on:
      - heavyaiserver

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

      # Map this volume if binary encryption mode is configured and certificates are being validated
      # - /var/lib/heavyai/cacerts.crt:/heavyai-storage/cacerts.crt

    environment:
      ### Required settings ###

      OMNISCIDB_CONTAINER_NAME: heavyaiserver
      SPAWNER_ENV_OMNISCI_HOST: heavyaiserver

      DOCKER_JUPYTER_LAB_IMAGE: *jlabimage

      OMNISCI_HOST: heavyaiserver
      OMNISCI_JUPYTER_ROLE: heavyai_jupyter
      JLAB_DOCKER_NETWORK_NAME: heavyai-backend

      ### Optional settings  ###

      # DOCKER_NOTEBOOK_DIR: /home/jovyan                   # The directory inside the user's Jupyter Lab container to mount the user volume to.
      # HUB_IP: jupyterhub                                  # The hostname or IP of the Jupyter Hub server
      # JHUB_BASE_URL: /jupyter/                            # The base URL prepended to all Jupyter Hub and Lab requests
      # HEAVYAI_ALLOW_SUPERUSER_ROLE: "false"               # Enable / disable admin access to Jupyter Hub
      # JLAB_CONTAINER_AUTOREMOVE: "true"                   # Enable / disable automatic removal of stopped Juptyer Lab containers
      # JLAB_DOCKER_NETWORK_NAME: heavyai-backend           # The docker network name for Jupyter Lab containers
      # JLAB_IDLE_TIMEOUT: 3600                             # Shut down Jupyter Lab containers after this many seconds of idle time
      # JLAB_NAME_PREFIX: jupyterlab                        # Container name prefix for Lab containers
      # JLAB_HEAVYAI_IMPORT_VOLUME_PATH: /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox  # Local Docker host path for where to mount the shared directory available to the HeavyDB server for file ingest
      # JLAB_NOTEBOOK_TERMINALS_ENBLED: "false"             # Enable terminals in notebooks
      # JLAB_USER_VOLUME_PATH: /var/lib/heavyai/jupyter/    # Local Docker host path to be used for user Jupyter Lab home directory volume mapping
      # JUPYTER_DEBUG: "true"                               # Turn on / off debugging for Jupyter Hub and Lab
      # OMNISCI_BINARY_TLS_CACERTS_LOCAL_PATH: /var/lib/heavyai/cacerts.crt # Specifying this or mapping a volume in the Hub container to /heavyai-storage/cacerts.crt will automatically enable binary TLS mode
      # OMNISCI_BINARY_TLS_VALIDATE: "false" # Whether or not to validate certificates in binary TLS mode. Specifying either "true" or "false" will enable binary TLS mode
      # OMNISCI_PORT: 6278 # Port that Jupyter Hub should use to connect to HEAVY.AI. Ensure this matches the protocol
      # OMNISCI_PROTOCOL: http # Protocol that Jupyter Hub should use to connect to HEAVY.AI. Ensure this is "binary" if using binary encryption
      # SPAWNER_CPU_LIMIT: 1                                # Number of CPU cores available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.cpu_limit
      # SPAWNER_RAM_LIMIT: 10G                              # Amount of CPU RAM available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.LocalProcessSpawner.mem_limit
      # SPAWNER_ENV_OMNISCI_HOST: heavyaiserver             # Hostname / IP address of the HEAVY.AI server for Lab containers to connect to by default
      # SPAWNER_ENV_OMNISCI_PORT: 6274                      # Port of the HEAVY.AI server for Lab containers to connect to by default
      # SPAWNER_ENV_OMNISCI_PROTOCOL: binary                # Protocol of the HEAVY.AI server for Lab containers to connect to by default
      # OMNISCI_DB_URL: "heavyai://heavyaiserver:6274/heavyai" # Alternative, direct connection (not Immerse session) to heavyaiserver. Username will be inferred, but password will be required in notebook.

      # More volumes for lab containers:
      # JLAB_VOLUME_1: /data1:/data1:rw
      # JLAB_VOLUME_2: /var/lib/heavyai:/heavyai-storage:ro


networks:
  heavyai-frontend:
    driver: bridge
    name: heavyai-frontend
  heavyai-backend:
    driver: bridge
    name: heavyai-backend

we are modìifying the docs as a first step this way.

Many thanks for you precious contribution

Candido

Gianfranco Campana

October 07, 2022 08:27
Hi Candido, just to let you know that with
```
OMNISCI_JUPYTER_ROLE: heavyai_jupyter ```

assigning heavyai_jupyter role to the user, the Jupyter button disappears, while with :
```
OMNISCI_JUPYTER_ROLE: omnisci_jupyter```

assigning omnisci_jupyter role to the user, it is shown.
0

Comment actions Permalink
Candido Dessanti

October 07, 2022 14:04
Hi Gianfranco,

I'll check it out because I think I have used the heavyai_jupyter role for the user. I'll back to you when checked, in the meantime thanks for the help.

Candido
0

Comment actions Permalink

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?