Docker and Data

Comments

11 comments

  • Avatar
    Candido Dessanti

    Hi @Ivan1981,

    the database, catalogs, and so on are located in the host filesystem that's outside docker image, so if you drop a container and you create a new one you have the same data (tables, dashboards, users, and whatever) you had in the previously created container.

    Can you share the docker parameter you are using to run heavyai?

    For an example this one

    -v /var/lib/heavyai:/var/lib/heavyai

    is using the /var/lib/heavyai in the localhost as storage for the data, so if drop and re-create the container the data will be there.

    sudo docker run -d --gpus=all \\
    -v /var/lib/heavyai:/var/lib/heavyai \\
    -p 6273-6278:6273-6278 \\
    heavyai/heavyai-ee-cuda:latest
    

    Besta Regards, Candido

    0
    Comment actions Permalink
  • Avatar
    Ivan1981

    Hello! I do everything according to the instructions.

    https://docs.heavy.ai/installation-and-configuration/installation/install-docker/docker-enterprise-edition-gpu#heavy.ai-installation

    The first container that is started actually stores data in the source directory, such as the "test" database. After deleting the container and starting a new one, there is no connection to the source directory, the "test" database is missing

    0
    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Hi,

    if you are following the instruction on our site your data should be stored into

    /var/lib/heavyai/storage/data

    If you create a new database named test and using it a table called test you should have in the data directory a directory containing the table named /var/lib/heavyai/storage/data/table_3_1

    If everything is correctly configured, after the restart of the container (or a drop and a re-create of the container with a different version for that matter) the directory should be there.

    If not is because everything has been written inside the container, so by destroying it the data will be lost, but this can happens just in the case that the bind volume mount is failing for some reason.

    To be sure that everything is ok, check the presence of files in the host filesystem.

    Best Regards, Candido

    0
    Comment actions Permalink
  • Avatar
    Иван Л

    Hey! I follow the instructions, I see the installation of the folder
    inside heavyai
    there are folders in it storage
    in the spreadsheet folder
    log
    lockfiles
    export
    disk_cache
    data
    catalogs
    Docker autostarting a container, but the container is shutting down with an error

    startheavy 8 running

    Backend TCP:  localhost:6274

    Backend HTTP: localhost:6278

    Frontend Web: localhost:6273

    Calcite TCP:  localhost:6279

    Warning: Cannot find the Nvidia Vulkan driver manifest file "nvidia_icd.json" in the expected system directories. As a result the backend renderer may not work.

      Please verify that the nvidia driver and vulkan loader are installed appropriately.

      See: https://docs.omnisci.com/troubleshooting/vulkan-graphics-api-beta#bare-metal-installs for some installation and troubleshooting tips.

    - heavydb 19 started

    - heavy_web_server 22 started

    Navigate to: http://localhost:6273

    startheavy 8 exited

    time="2023-01-14T23:40:29Z" level=fatal msg="Error reading config file: While parsing config: (19, 22): keys cannot contain new lines"

    Usage Error: unrecognised option 'flush-log'

     

    0
    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Can you post the contents of heavy.conf?

    It seems that there is a problem with that.

    0
    Comment actions Permalink
  • Avatar
    Иван Л

    Hi,

    I found a connection issue. I started the container from the docker GUI, I realized that it is correct to run through the terminal using the command:

    sudo docker run -d \
    -v /var/lib/heavyai:/var/lib/heavyai \
    -p 6273-6278:6273-6278 \
    heavyai/heavyai-ee-cpu:latest

    0
    Comment actions Permalink
  • Avatar
    Иван Л

    Help solve the problem of getting data through the Heavy Connector from the S3 server or a local server.
    From the local computer, I can easily upload files CSV
        however when i use your file at https://omnisci-data-catalog-public-usw2.s3-us-west-2.amazonaws.com/us_states/tables/us_states/data/us_states.zip
    when loading it i see this error

    File or directory "/var/lib/heavyai/storage/import/1d9c63698a6e2d90c348df88b22cd99ff61b7b48f15bd87b3fd7aa2ab0b6628a/us_states.zip" does not exist.

    She repeats every time. Can you reproduce this error yourself?

    0
    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Hi,

     

    could you share the steps you are doing to get the error? I tried to load the s3 publich file with you link using a docker instance running with the root user and I didn't get any issue.

    in the meantime do these checks:

    1. Run the ps command to be sure that the web server and the database server are running using the same storage directory (so the same import directory)
      mapd@zion-tr:/opt/mapd_storage/github/master2_test/heavydb-internal/build$ ps -ef | egrep "heavydb|heavy_web_server"
      root      545242  545236  5 11:56 ?        00:00:37 ./bin/heavydb /var/lib/heavyai/storage --port 6274 --http-port 6278 --calcite-port 6279 --config /var/lib/heavyai/heavy.conf
      root      545245  545236  0 11:56 ?        00:00:06 ./bin/heavy_web_server --port 6273 --backend-url http://localhost:6278 --data /var/lib/heavyai/storage --config /var/lib/heavyai/heavy.conf
       check for the flag --config into the result of the command
    2. Check that the user you are running the database and the web server has the right permission to read and write in the filesystem where the storage directory is (you can get the user and the storage directory using the previous command) and issuing a list command
      ls -la /var/lib/heavyai/storage/import/
      total 16
      drwxrwx---  4 root mapd 4096 gen 23 11:59 .
      and check if the user that's running the software can access the directory and that your containers are using that host directory running the docker command with  the inspect switch
      docker inspect -f '{{ .Mounts }}' [your container id]
      [{bind  /var/lib/heavyai /var/lib/heavyai   true rprivate}

    Using a regular docker container, going in Immerse on data panel, and navigating thru

    "Add Table"'->"Amazon S3"->"s3 Link" and then using the link you provided and Importing data, works for me, but I'm not sure if I'm doing exactly the same steps as yours.
    I can reproduce the same error using different paths for storage data of the web server/database or running with different users; this already happened in the past with a different user, and for this reason, I asked you to run the previous commands.

    I hope to hear from you soon.

    regards,
    Candido

     

    0
    Comment actions Permalink
  • Avatar
    Иван Л

    $ ps -ef | egrep "heavydb|heavy_web_server"

      501 74029 73981   0  9:18   ttys000    0:00.01 egrep heavydb|heavy_web_server

     

    $ ls -la /var/lib/heavyai/storage/import/

    total 0

    drwxr-x---   3 ivanzaharov  wheel   96 24 янв 09:12 .

    drwxr-xr-x  16 ivanzaharov  wheel  512 24 янв 09:12 ..

    drwxr-x---   3 ivanzaharov  wheel   96 24 янв 09:12 e5b40fc2f9bb2edd8ef6e41f30d8807299172469066f0efbe0fe0681c68fb211

     

    $ docker inspect -f '{{ .Mounts }}' 27aa8945d870

    [{bind  /var/lib/heavyai /var/lib/heavyai   true rprivate}]

    0
    Comment actions Permalink
  • Avatar
    Candido Dessanti

    Hi,

     

    you should run the ps -ef | egrep "heavydb|heavy_web_server" when the container/containers is/are running

    0
    Comment actions Permalink
  • Avatar
    Иван Л

     

    container is running

    MacBook-Air-Ivan:~ ivanzaharov$ docker ps

    CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS         PORTS                              NAMES

    8523788d0c10   heavyai/heavyai-ee-cpu:latest   "/bin/sh -c '/opt/he…"   34 minutes ago   Up 3 minutes   0.0.0.0:6273-6278->6273-6278/tcp   festive_jang

    MacBook-Air-Ivan:~ ivanzaharov$ ps -ef | egrep "heavydb|heavy_web_server"

      501   675   431   0 12:41   ttys000    0:00.00 egrep heavydb|heavy_web_server

     

    0
    Comment actions Permalink

Please sign in to leave a comment.