Jupyterlab not working on clean Ubuntu 20.04 LTE
Just installed a clean Ubuntu 20.04 LTE with heavyai 6.1.1 docker gpu free edition, and jupyterlab.
The problem is Jupyterlab container is always in "exited" status (it seems not normal to me) , and there isn't any Jupiter button in Immerse or Heavysql Editor.
Docker and nvidia cuda are working fine, here some detail of this installation:
nvidia cuda:
/var/lib/heavyai$ docker run --rm nvidia/cudagl:11.0-runtime-ubuntu18.04 nvidia-smi
Mon Aug 29 11:47:13 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 28% 58C P2 43W / 180W | 1163MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
docker container with exited jupyterlab:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7006f54fde0f omnisci/omnisci-jupyterhub:v0.4 "/docker-entrypoint.…" 3 days ago Up About an hour 8000/tcp jupyterhub
8b9874edefdd heavyai/heavyai-ee-cuda:latest "/bin/sh -c '/opt/he…" 4 days ago Up About an hour 0.0.0.0:6273-6274->6273-6274/tcp, :::6273-6274->6273-6274/tcp, 0.0.0.0:6278->6278/tcp, :::6278->6278/tcp heavyaiserver
aabd5c89e436 omnisci/omnisci-jupyterlab-cpu:v0.5 "tini -g -- echo" 4 days ago Exited (0) About an hour ago heavyai_jupyterlab-tmp_1
docker-compose.yml.txt|attachment (5.1 KB) heavyai.conf.txt|attachment (92 Bytes) servers.json.txt|attachment (38 Bytes)
User and role created and assigned:
heavysql> \
oles
heavyai_jupyter
heavysql> \
ole_list user1
heavyai_jupyter
folders tree:
WS:/var/lib$
gc@gc-Z10PE-D16-WS:/var/lib$ ls -la |grep heavyai
drwxr-xr-x 5 gc root 4096 ago 29 14:04 heavyai
gc@gc-Z10PE-D16-WS:/var/lib$ cd heavyai
gc@gc-Z10PE-D16-WS:/var/lib/heavyai$ ls -la
total 36
drwxr-xr-x 5 gc root 4096 ago 29 14:04 .
drwxr-xr-x 75 root root 4096 ago 24 11:30 ..
drwxr-xr-x 3 root root 4096 ago 25 12:33 data
-rw-rw-r-- 1 gc gc 5265 ago 29 12:34 docker-compose.yml
-rw-rw-r-- 1 gc gc 92 ago 29 12:27 heavyai.conf
drwxr-xr-x 2 root root 4096 ago 24 13:20 jupyter
-rw-rw-r-- 1 gc gc 38 ago 25 15:44 servers.json
drwxr-xr-x 10 root root 4096 ago 24 13:08 storage
gc@gc-Z10PE-D16-WS:/var/lib/heavyai$ cd jupyter
gc@gc-Z10PE-D16-WS:/var/lib/heavyai/jupyter$ ls -la
total 8
drwxr-xr-x 2 root root 4096 ago 24 13:20 .
drwxr-xr-x 5 gc root 4096 ago 29 14:04 ..
gc@gc-Z10PE-D16-WS:/var/lib/heavyai/jupyter$
The Jupyterlab container log is empty:
WS:/var/lib/heavyai/jupyter$ docker logs --tail 50 --follow --timestamps aabd5c89e436
2022-08-25T09:56:07.857247368Z
2022-08-25T10:03:21.664031064Z
2022-08-25T10:06:12.291297711Z
2022-08-25T10:36:50.836189185Z
2022-08-25T10:40:12.694607414Z
2022-08-25T10:49:16.171701240Z
2022-08-25T10:50:03.308175355Z
2022-08-25T11:07:19.288349578Z
2022-08-25T11:29:24.304789204Z
2022-08-25T11:44:55.033777252Z
2022-08-25T13:13:37.071985704Z
2022-08-25T13:25:59.243723916Z
2022-08-25T13:39:00.748249237Z
2022-08-29T10:34:57.008028066Z
gcampana@gcampana-Z10PE-D16-WS:/var/lib/heavyai/jupyter$
This is a clean Ubuntu 20.04 image and straightforward Docker/heavyai/Jupyter installation, what should I still check ?
Any help ?
-
Hi Gianfranco,
I've made some revisions to your docker-compose file and attached them here. It looks like you're not overriding the default start command, and thus the default start command would expect for a configuration file to be at /var/lib/heavyai/heavy.conf not /heavyai-storage/heavyai.conf, and further for your data folder within the container to exist at /var/lib/heavyai/.
Additionally, the environmental variables used by the jupyterhub/jupyterlab containers haven't been updated, so I've ammended these as well.
On our end, it looks like our jupyter install page needs an update; sorry for the inconvenience.
heavy.conf.txt|attachment (92 Bytes) docker-compose.yml.revised.txt|attachment (5.1 KB)
Hope this helps! Look forward to your feedback.
Thanks, Neill
-
Thanks for your help Neil, using your files improved the situation.
Now I can see the Jupyter button with the "admin" user (not valid for jupyerlab access because I can't assign roles to admin user), but I don't see the Jupyterlab button with the "user1" user, to which I assigned the heavyai_jupyter role and all privileges on database:
heavysql> grant all on database heavyai to user1; heavysql> \ ole_list user1 heavyai_jupyterIn addition, often the http sessions don't disconnect and are left active: here I have two http sessions active, but I don't know how to re-enter properly in the browser in order to log-out , or how to kill (stop) the session in order to free the session.
heavysql> show user sessions ; session_id|login_name|client_address|db_name 321-9azc|admin|http:localhost:48732|heavyai 428-R1kc|admin|tcp:localhost:56668|heavyai 838-kdaT|user1|http:172.19.0.1|heavyaiIf I try to re-enter the session in browser with both "admin" and "user1" login, I get:
.and I imagine hit this error could be the cause the session is left active and pending. What does "JSON.parse: unexpected end of data at line 1 column 1 of the JSON data" exactly mean, and what should I do to avoid that error?
I added a ten minutes timeout in heavyai.conf but it would be nice to manage (and kill) sessions if needed: from what moment will this parameter (and in general parameters edited in heavyai.conf) be active?
idle-session-duration 10Update: after stopping and restarting heavyai container the Jupyter button is missing with both users admin and user1: it seems that the "JSON.parse:" error is gone, now.
-
Hi Gianfranco,
Now I can see the Jupyter button with the “admin” user (not valid for jupyerlab access because I can’t assign roles to admin user), but I don’t see the Jupyterlab button with the “user1” user, to which I assigned the heavyai_jupyter role and all privileges on database:
Can you please try to use the role "omnisci_jupyter" instead and also make this change in docker-compose.yml?
Additionally, under optional settings, try to add this line to allow your admin user access:
OMNISCI_ALLOW_SUPERUSER_ROLE: "true"(note rename from HEAVYAI_ to OMNISCI_ if uncommenting existing line).Note that a full
docker compose up -dis required for changes in docker-compose.yml to be enabled.From what moment will this parameter (and in general parameters edited in heavyai.conf) be active?
These settings require a restart of heavydb and/or heavy_web_server.
-
HI Neil, I followed your hints but I still have no Jupyterlab button in Immerse with both the user "Admin" and "user1". Role omnisci_jupyter is created and granted, and OMNISCI_ALLOW_SUPERUSER_ROLE: "true" is set and activated with compose up -d and restart of heavydb and heavy_web_server.
heavysql> \ ole_list user1 heavyai_jupyter omnisci_jupyterdocker-compose.yml.txt|attachment (5.2 KB) heavyai.conf.txt|attachment (117 Bytes) servers.json.txt|attachment (38 Bytes)
This is the heavyai cotainer log, I see something strange:
2022-09-01T07:05:46.529369127Z - heavydb 17 started 2022-09-01T07:05:46.529950585Z - heavy_web_server 18 started 2022-09-01T07:05:49.382755191Z ⇨ http server started on [::]:6273 2022-09-01T07:05:51.532355298Z Navigate to: http://localhost:6273 2022-09-01T07:13:30.556937397Z Failed to write to log, write /var/lib/heavyai/storage/log/heavy_web_server.37460dcae923.root.log.ALL.20220901-070546.18: file already closed 2022-09-01T07:13:30.666992326Z 2022-09-01T07:13:30.666678 E 17 0 7 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278 2022-09-01T07:13:30.667022606Z 2022-09-01T07:13:30.666762 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276 2022-09-01T07:13:30.667093997Z 2022-09-01T07:13:30.666719 E 17 0 8 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274 2022-09-01T07:13:32.561755610Z startheavy 7 exited 2022-09-01T07:13:34.301915394Z startheavy 7 running 2022-09-01T07:13:34.312219693Z WARN: config file does not exist, ignoring: --config /var/lib/heavyai/heavy.conf 2022-09-01T07:13:34.312290332Z Backend TCP: localhost:6274 2022-09-01T07:13:34.312313508Z Backend HTTP: localhost:6278 2022-09-01T07:13:34.312319797Z Frontend Web: localhost:6273 2022-09-01T07:13:34.312325600Z Calcite TCP: localhost:6279It is searching /var/lib/heavyai/heavy.conf, this seems an error
-
I observer in dockerfile.cuda the startheavy command calling heavy.conf insetad of heavyai.conf
CMD /opt/heavyai/startheavy --non-interactive --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.confand also in the container log the entry:
2022-09-01T12:32:50.948375791Z WARN: config file does not exist, ignoring: --config /var/lib/heavyai/heavy.confso I copied my heavyai.conf as heavy.conf and the Jupyter button magically appeared ^_^
Now, (hopefully) I need a last step in order to complete the installation: opening Jupyter from the user "user1" leads to :
but actually the omnisci_jupyter_role is granted to that user:
heavysql> \ ole_list user1 heavyai_jupyter omnisci_jupyter heavysql>Any help for the role issue?
-
Well, i renamed the parameter name in the yml file and now the role is found on the user. But a different error is returned in Jupyterlab :
The user "user1" is already connected to Heavyai and Immerse, therefore it isn't an authentication problem.
I still have this error in the container log, this could be the cause:
2022-09-02T14:07:47.911421271Z 2022-09-02T14:07:47.911066 E 17 0 9 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6278 2022-09-02T14:07:47.911517968Z 2022-09-02T14:07:47.911094 E 17 0 10 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6274 2022-09-02T14:07:47.911528142Z 2022-09-02T14:07:47.911118 E 17 0 11 HeavyDB.cpp:194 Exception: Thrift server exited: Cannot assign requested address: port 6276
docker-compose.yml.txt|attachment (5.3 KB) heavy.conf.txt|attachment (119 Bytes) heavyai.conf.txt|attachment (119 Bytes) servers.json.txt|attachment (38 Bytes)
-
I marked the preceding post a Solution, because the original issue (missing Jupyter button) is solved, by setting the new parameter role and parameter value in docker-compose.yml and granting that to the user:
OMNISCI_JUPYTER_ROLE: omnisci_jupyterand using the " docker-compose.yml.revised.txt|attachment"
I will open a new thread for the different error "[Errno 99] Cannot assign requested address: port xxxx".
Thanks Neill.
-
[quote="Neill, post:2, topic:3044"] Hi Neill, [/quote]
could you just explain better what does it mean " overriding the default start command " , please?
Would I have to change something else respect to what is reported into the documentation, or I missed something already reported here ?
Could you point me to where are documented the details about "overriding the default start command" ?
Please sign in to leave a comment.
Comments
9 comments