Docker and Data
Docker creates a container, I create a table and a dashboard. When I delete a container and create a new one with the same port, I don't see the table and dashboard. How to reconnect
-
Hi @Ivan1981,
the database, catalogs, and so on are located in the host filesystem that's outside docker image, so if you drop a container and you create a new one you have the same data (tables, dashboards, users, and whatever) you had in the previously created container.
Can you share the docker parameter you are using to run heavyai?
For an example this one
-v /var/lib/heavyai:/var/lib/heavyai
is using the /var/lib/heavyai in the localhost as storage for the data, so if drop and re-create the container the data will be there.
sudo docker run -d --gpus=all \\ -v /var/lib/heavyai:/var/lib/heavyai \\ -p 6273-6278:6273-6278 \\ heavyai/heavyai-ee-cuda:latestBesta Regards, Candido
-
Hello! I do everything according to the instructions.
The first container that is started actually stores data in the source directory, such as the "test" database. After deleting the container and starting a new one, there is no connection to the source directory, the "test" database is missing
-
Hi,
if you are following the instruction on our site your data should be stored into
/var/lib/heavyai/storage/data
If you create a new database named test and using it a table called test you should have in the data directory a directory containing the table named
/var/lib/heavyai/storage/data/table_3_1If everything is correctly configured, after the restart of the container (or a drop and a re-create of the container with a different version for that matter) the directory should be there.
If not is because everything has been written inside the container, so by destroying it the data will be lost, but this can happens just in the case that the bind volume mount is failing for some reason.
To be sure that everything is ok, check the presence of files in the host filesystem.
Best Regards, Candido
-
Hey! I follow the instructions, I see the installation of the folder
inside heavyai
there are folders in it storage
in the spreadsheet folder
log
lockfiles
export
disk_cache
data
catalogs
Docker autostarting a container, but the container is shutting down with an errorstartheavy 8 runningBackend TCP: localhost:6274Backend HTTP: localhost:6278Frontend Web: localhost:6273Calcite TCP: localhost:6279Warning: Cannot find the Nvidia Vulkan driver manifest file "nvidia_icd.json" in the expected system directories. As a result the backend renderer may not work.Please verify that the nvidia driver and vulkan loader are installed appropriately.See: https://docs.omnisci.com/troubleshooting/vulkan-graphics-api-beta#bare-metal-installs for some installation and troubleshooting tips.- heavydb 19 started- heavy_web_server 22 startedNavigate to: http://localhost:6273startheavy 8 exitedtime="2023-01-14T23:40:29Z" level=fatal msg="Error reading config file: While parsing config: (19, 22): keys cannot contain new lines"Usage Error: unrecognised option 'flush-log' -
Help solve the problem of getting data through the Heavy Connector from the S3 server or a local server.
From the local computer, I can easily upload files CSV
however when i use your file at https://omnisci-data-catalog-public-usw2.s3-us-west-2.amazonaws.com/us_states/tables/us_states/data/us_states.zip
when loading it i see this errorFile or directory "/var/lib/heavyai/storage/import/1d9c63698a6e2d90c348df88b22cd99ff61b7b48f15bd87b3fd7aa2ab0b6628a/us_states.zip" does not exist.
She repeats every time. Can you reproduce this error yourself?
-
Hi,
could you share the steps you are doing to get the error? I tried to load the s3 publich file with you link using a docker instance running with the root user and I didn't get any issue.
in the meantime do these checks:- Run the ps command to be sure that the web server and the database server are running using the same storage directory (so the same import directory)
mapd@zion-tr:/opt/mapd_storage/github/master2_test/heavydb-internal/build$ ps -ef | egrep "heavydb|heavy_web_server"
check for the flag --config into the result of the command
root 545242 545236 5 11:56 ? 00:00:37 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
root 545245 545236 0 11:56 ? 00:00:06 ./bin/heavy_web_server --port 6273 --backend-url http://localhost:6278 --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf - Check that the user you are running the database and the web server has the right permission to read and write in the filesystem where the storage directory is (you can get the user and the storage directory using the previous command) and issuing a list command
ls -la /var/lib/heavyai/storage/import/
and check if the user that's running the software can access the directory and that your containers are using that host directory running the docker command with the inspect switch
total 16
drwxrwx--- 4 root mapd 4096 gen 23 11:59 .
docker inspect -f '{{ .Mounts }}' [your container id]
[{bind /var/lib/heavyai /var/lib/heavyai true rprivate}
Using a regular docker container, going in Immerse on data panel, and navigating thru
"Add Table"'->"Amazon S3"->"s3 Link" and then using the link you provided and Importing data, works for me, but I'm not sure if I'm doing exactly the same steps as yours.
I can reproduce the same error using different paths for storage data of the web server/database or running with different users; this already happened in the past with a different user, and for this reason, I asked you to run the previous commands.
I hope to hear from you soon.
regards,
Candido - Run the ps command to be sure that the web server and the database server are running using the same storage directory (so the same import directory)
-
$ ps -ef | egrep "heavydb|heavy_web_server"
501 74029 73981 0 9:18 ttys000 0:00.01 egrep heavydb|heavy_web_server
$ ls -la /var/lib/heavyai/storage/import/
total 0
drwxr-x--- 3 ivanzaharov wheel 96 24 янв 09:12 .
drwxr-xr-x 16 ivanzaharov wheel 512 24 янв 09:12 ..
drwxr-x--- 3 ivanzaharov wheel 96 24 янв 09:12 e5b40fc2f9bb2edd8ef6e41f30d8807299172469066f0efbe0fe0681c68fb211
$ docker inspect -f '{{ .Mounts }}' 27aa8945d870
[{bind /var/lib/heavyai /var/lib/heavyai true rprivate}]
-
container is running
MacBook-Air-Ivan:~ ivanzaharov$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8523788d0c10 heavyai/heavyai-ee-cpu:latest "/bin/sh -c '/opt/he…" 34 minutes ago Up 3 minutes 0.0.0.0:6273-6278->6273-6278/tcp festive_jang
MacBook-Air-Ivan:~ ivanzaharov$ ps -ef | egrep "heavydb|heavy_web_server"
501 675 431 0 12:41 ttys000 0:00.00 egrep heavydb|heavy_web_server
-
AH, probably in the mac, the processes running on docker aren't showing up.
So please connect to the running container and run the command inside of it (you have to change the id)
docker exec -it 8523788d0c10 bash
root@8523788d0c10:/opt/heavyai# ps -ef | egrep "heavydb"
root 18 12 0 09:37 ? 00:00:00 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
root 135 124 0 09:47 pts/0 00:00:00 grep -E --color=auto heavydb
root@ee4672c0db87:/opt/heavyai# ps -ef | egrep "heavydb|heavy_web_server"
root 18 12 0 09:37 ? 00:00:00 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
root 21 12 0 09:37 ? 00:00:04 ./bin/heavy_web_server --port 6273 --backend-url http://localhost:6278 --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf
root 137 124 0 09:47 pts/0 00:00:00 grep -E --color=auto heavydb|heavy_web_serverIt would be helpful also have the web server and database logfiles that should be located into
/var/lib/heavyai/storage/log/
Regards,Candido
-
root@911336f786f6:/opt/heavyai# ps -ef | egrep "heavydb"
root 19 13 1 07:49 ? 00:00:13 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
root 102 89 0 08:02 pts/0 00:00:00 grep -E --color=auto heavydb
root@911336f786f6:/opt/heavyai# ps -ef | egrep "heavydb|heavy_web_server"
root 19 13 1 07:49 ? 00:00:13 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
root 22 13 2 07:49 ? 00:00:18 ./bin/heavy_web_server --port 6273 --backend-url http://localhost:6278 --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf
root 104 89 0 08:03 pts/0 00:00:00 grep -E --color=auto heavydb|heavy_web_server
root@911336f786f6:/opt/heavyai#
Please sign in to leave a comment.
Comments
14 comments