Jupyter installation: [Errno 99] Cannot assign requested address
I just installed and configured Docker/Jupyter/Heavyai on Ubuntu 20.04.
Heavydb and Immerse are working fine, and I can see the Jupyter button in Immerse.
But opening Jupyter return this error:
401 : Unauthorized
Error with OmniSci authentication: [Errno 99] Cannot assign requested address
I can see the same error also in the container log:
2022-09-05T08:36:52.345705462Z 2022-09-05T08:36:52.345389 E 17 0 7 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278
2022-09-05T08:36:52.345742561Z 2022-09-05T08:36:52.345530 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274
2022-09-05T08:36:52.345926321Z 2022-09-05T08:36:52.345464 E 17 0 8 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276
here the configuration file currently used: docker-compose.yml.txt|attachment (5.2 KB) heavyai.conf.txt|attachment (119 Bytes) servers.json.txt|attachment (38 Bytes)
-
After a ton of testing and debugging i got Docker/Jupyter/heavy db working: I used a number of configuration parameters that are different from what is documented in online docs.
Starting from the revision made in docker-compose.yml revised shared by Neill in this post , I also modified in docker-compose.yml:
- heavyaiserver: container_name: heavyaiserver --> omnisciserver
-
HEAVYAI_HOST: heavyaiserver --> OMNISCI_HOST: heavyaiserver
-
\#JLAB_DOCKER_NETWORK_NAME: heavyai-backend --> uncomment this
With these parameters in place the notebook is spawned correctly:
I thought it might be useful for other users out there ;)
-
Here the containers full command exposing the ports:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3a7ea963f8256d97fdf563223426af70b382d7bef9806d9dccf813d97cdb5128 omnisci/omnisci-jupyterhub:v0.4 "/docker-entrypoint.sh jupyterhub -f /srv/jupyterhub/jupyterhub_config.py" 10 minutes ago Up 10 minutes 8000/tcp jupyterhub d1a1671cff2f96322bcdd00030f3d2abc129fa9097ac27bb922148e2819483c0 omnisci/omnisci-jupyterlab-cpu:v0.5 "tini -g -- echo" 10 minutes ago Exited (0) 10 minutes ago heavyai_jupyterlab-tmp_1 da84e0f600e13f23bc63c1f3752c5b3621115bb51a3dcf5207d835f7ac5a345f heavyai/heavyai-ee-cuda:latest "/bin/sh -c '/opt/heavyai/startheavy --non-interactive --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf'" 10 minutes ago Up 10 minutes 0.0.0.0:6273-6274->6273-6274/tcp, :::6273-6274->6273-6274/tcp, 0.0.0.0:6278->6278/tcp, :::6278->6278/tcp heavyaiserver
Here the jupyrterhub container log:
docker logs --tail 250 --follow --timestamps 3a7ea963f825 2022-09-08T15:40:57.275859002Z Generating a RSA private key 2022-09-08T15:40:57.952654793Z .................................................................................................................................................++++ 2022-09-08T15:40:57.969533364Z ..++++ 2022-09-08T15:40:57.970084383Z writing new private key to '/etc/ssl/private/jhubcert.key' 2022-09-08T15:40:57.970499210Z ----- 2022-09-08T15:40:59.357718145Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Authenticator: jhub-omnisci-authenticator.omnisci_user_auth.OmniSciUserAuthenticator 2022-09-08T15:40:59.357749367Z [I 2022-09-08 15:40:59.357 JupyterHub app:2120] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1 2022-09-08T15:40:59.362819375Z [I 2022-09-08 15:40:59.362 JupyterHub app:1302] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret 2022-09-08T15:40:59.386177869Z [I 2022-09-08 15:40:59.385 alembic.runtime.migration migration:130] Context impl SQLiteImpl. 2022-09-08T15:40:59.386205151Z [I 2022-09-08 15:40:59.386 alembic.runtime.migration migration:137] Will assume non-transactional DDL. 2022-09-08T15:40:59.421206486Z [I 2022-09-08 15:40:59.420 alembic.runtime.migration migration:356] Running stamp_revision -> 4dc2d5a8c53c 2022-09-08T15:40:59.789274724Z [I 2022-09-08 15:40:59.788 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN 2022-09-08T15:40:59.817261218Z [I 2022-09-08 15:40:59.817 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed. 2022-09-08T15:40:59.943050210Z [I 2022-09-08 15:40:59.942 JupyterHub app:2337] Hub API listening on http://172.19.0.3:8081/jupyter/hub/ 2022-09-08T15:40:59.944336999Z [W 2022-09-08 15:40:59.944 JupyterHub proxy:642] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else... 2022-09-08T15:40:59.944361212Z [I 2022-09-08 15:40:59.944 JupyterHub proxy:645] Starting proxy @ http://:8000/jupyter/ 2022-09-08T15:41:00.554298654Z 15:41:00.552 - info: [ConfigProxy] Proxying http://*:8000 to (no default) 2022-09-08T15:41:00.554783154Z 15:41:00.554 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes 2022-09-08T15:41:00.583033824Z [I 2022-09-08 15:41:00.582 JupyterHub app:2362] Starting managed service cull_idle 2022-09-08T15:41:00.583420371Z [I 2022-09-08 15:41:00.583 JupyterHub service:316] Starting service 'cull_idle': ['python', '/srv/jupyterhub/cull_idle_servers.py', '--timeout=3600'] 2022-09-08T15:41:00.590010542Z [I 2022-09-08 15:41:00.589 JupyterHub service:121] Spawning python /srv/jupyterhub/cull_idle_servers.py --timeout=3600 2022-09-08T15:41:00.606208627Z [I 2022-09-08 15:41:00.605 JupyterHub proxy:319] Checking routes 2022-09-08T15:41:00.606238386Z [I 2022-09-08 15:41:00.606 JupyterHub proxy:399] Adding default route for Hub: /jupyter/ => http://172.19.0.3:8081 2022-09-08T15:41:00.609329584Z [I 2022-09-08 15:41:00.609 JupyterHub app:2422] JupyterHub is now running at http://:8000/jupyter/ 2022-09-08T15:41:00.808231000Z [I 2022-09-08 15:41:00.807 JupyterHub log:174] 200 GET /jupyter/hub/api/users (cull_idle@172.19.0.3) 40.23ms 2022-09-08T15:42:18.743364593Z [E 2022-09-08 15:42:18.742 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address 2022-09-08T15:42:18.743437615Z [W 2022-09-08 15:42:18.743 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true (172.19.0.1): Error with OmniSci authentication: [Errno 99] Cannot assign requested address 2022-09-08T15:42:18.794422406Z [W 2022-09-08 15:42:18.794 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@172.19.0.1) 54.86ms 2022-09-08T15:42:33.943313935Z [E 2022-09-08 15:42:33.942 JupyterHub omnisci_user_auth:111] Error with OmniSci authentication: [Errno 99] Cannot assign requested address 2022-09-08T15:42:33.943363026Z [W 2022-09-08 15:42:33.943 JupyterHub web:1782] 401 GET /jupyter/hub/login?newnotebook=true (172.19.0.1): Error with OmniSci authentication: [Errno 99] Cannot assign requested address 2022-09-08T15:42:33.946198452Z [W 2022-09-08 15:42:33.945 JupyterHub log:174] 401 GET /jupyter/hub/login?newnotebook=true (@172.19.0.1) 5.52ms
-
Any help? Anyone got the same problem installing Jupyter for Immerse?
This is still happening also after reinstalled Ubuntu, same identical error, and therefore the problem is easily reproducible with the configuration I'm using.In addition there is an open point about " overriding the default start command " since when this problem appeared :
There is a best practice I missed? Maybe with a different configuration the problem would not appear?
-
Hi @Gianfranco_Campana,
I suggest you leave the container names unchanged and do an override on defaults using this yaml file
version: '3.7' services: heavyaiserver: container_name: heavyaiserver image: heavyai/heavyai-ee-cuda:v6.0.0 restart: always ipc: shareable volumes: - /var/lib/heavyai:/var/lib/heavyai - /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox:/jhub_heavyai_dropbox networks: - heavyai-frontend - heavyai-backend ports: - "6273:6273" - "6274:6274" - "6278:6278" # If using binary encryption, uncomment the below line to override the default # command that uses startheavyai, noting that you must have an existing HEAVY.AI # data directory and run initdb before making this change. # command: /bin/bash -c "/heavyai/bin/heavy_web_server --config /heavyai-storage/heavyai.conf & /heavyai/bin/heavydb --config /heavyai-storage/heavyai.conf" # The purpose of this is to make sure the jlabimage is pulled because jupyterhub will not pull it automatically when launching jupyterlab-tmp: image: &jlabimage omnisci/omnisci-jupyterlab-cpu:v0.5 command: echo networks: - heavyai-backend jupyterhub: container_name: jupyterhub image: omnisci/omnisci-jupyterhub:v0.4 restart: always networks: - heavyai-backend depends_on: - heavyaiserver volumes: - /var/run/docker.sock:/var/run/docker.sock # Map this volume if binary encryption mode is configured and certificates are being validated # - /var/lib/heavyai/cacerts.crt:/heavyai-storage/cacerts.crt environment: ### Required settings ### OMNISCIDB_CONTAINER_NAME: heavyaiserver SPAWNER_ENV_OMNISCI_HOST: heavyaiserver DOCKER_JUPYTER_LAB_IMAGE: *jlabimage OMNISCI_HOST: heavyaiserver OMNISCI_JUPYTER_ROLE: heavyai_jupyter JLAB_DOCKER_NETWORK_NAME: heavyai-backend ### Optional settings ### # DOCKER_NOTEBOOK_DIR: /home/jovyan # The directory inside the user's Jupyter Lab container to mount the user volume to. # HUB_IP: jupyterhub # The hostname or IP of the Jupyter Hub server # JHUB_BASE_URL: /jupyter/ # The base URL prepended to all Jupyter Hub and Lab requests # HEAVYAI_ALLOW_SUPERUSER_ROLE: "false" # Enable / disable admin access to Jupyter Hub # JLAB_CONTAINER_AUTOREMOVE: "true" # Enable / disable automatic removal of stopped Juptyer Lab containers # JLAB_DOCKER_NETWORK_NAME: heavyai-backend # The docker network name for Jupyter Lab containers # JLAB_IDLE_TIMEOUT: 3600 # Shut down Jupyter Lab containers after this many seconds of idle time # JLAB_NAME_PREFIX: jupyterlab # Container name prefix for Lab containers # JLAB_HEAVYAI_IMPORT_VOLUME_PATH: /var/lib/heavyai/data/heavyai_import/jhub_heavyai_dropbox # Local Docker host path for where to mount the shared directory available to the HeavyDB server for file ingest # JLAB_NOTEBOOK_TERMINALS_ENBLED: "false" # Enable terminals in notebooks # JLAB_USER_VOLUME_PATH: /var/lib/heavyai/jupyter/ # Local Docker host path to be used for user Jupyter Lab home directory volume mapping # JUPYTER_DEBUG: "true" # Turn on / off debugging for Jupyter Hub and Lab # OMNISCI_BINARY_TLS_CACERTS_LOCAL_PATH: /var/lib/heavyai/cacerts.crt # Specifying this or mapping a volume in the Hub container to /heavyai-storage/cacerts.crt will automatically enable binary TLS mode # OMNISCI_BINARY_TLS_VALIDATE: "false" # Whether or not to validate certificates in binary TLS mode. Specifying either "true" or "false" will enable binary TLS mode # OMNISCI_PORT: 6278 # Port that Jupyter Hub should use to connect to HEAVY.AI. Ensure this matches the protocol # OMNISCI_PROTOCOL: http # Protocol that Jupyter Hub should use to connect to HEAVY.AI. Ensure this is "binary" if using binary encryption # SPAWNER_CPU_LIMIT: 1 # Number of CPU cores available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.cpu_limit # SPAWNER_RAM_LIMIT: 10G # Amount of CPU RAM available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.LocalProcessSpawner.mem_limit # SPAWNER_ENV_OMNISCI_HOST: heavyaiserver # Hostname / IP address of the HEAVY.AI server for Lab containers to connect to by default # SPAWNER_ENV_OMNISCI_PORT: 6274 # Port of the HEAVY.AI server for Lab containers to connect to by default # SPAWNER_ENV_OMNISCI_PROTOCOL: binary # Protocol of the HEAVY.AI server for Lab containers to connect to by default # OMNISCI_DB_URL: "heavyai://heavyaiserver:6274/heavyai" # Alternative, direct connection (not Immerse session) to heavyaiserver. Username will be inferred, but password will be required in notebook. # More volumes for lab containers: # JLAB_VOLUME_1: /data1:/data1:rw # JLAB_VOLUME_2: /var/lib/heavyai:/heavyai-storage:ro networks: heavyai-frontend: driver: bridge name: heavyai-frontend heavyai-backend: driver: bridge name: heavyai-backend
we are modìifying the docs as a first step this way.
Many thanks for you precious contribution
Candido
Please sign in to leave a comment.
Comments
6 comments