Lack of GPU Memory

March 16, 2021 22:16

I'm using Omnisci free GPU on Centos 8 GTX1080 8GB + 128GB RAM, but now I'm starting to experiment the first problems with the lack of memory: I have a pretty simple Dashboard with a table chart plus a clorophet chart that leads to this error:

Exception: OutOfGpuMemoryError: Cuda error code=801, CUDA_ERROR_UNKNOWN/CUDA_ERROR_NOT_SUPPORTED, possibly not enough gpu memory available for the requested buffer size of 1000000000 bytes on gpu 0.

And 7 out of 8 GB of GPU RAM already busy:

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 2386 C+G ...mnisci/bin/omnisci_server 6582MiB | | 0 N/A N/A 2890 G /usr/libexec/Xorg 40MiB | | 0 N/A N/A 3874 G /usr/bin/gnome-shell 45MiB | | 0 N/A N/A 6452 G /usr/libexec/Xorg 235MiB | | 0 N/A N/A 6569 G /usr/bin/gnome-shell 61MiB | | 0 N/A N/A 7831 G ...mviewer/tv_bin/TeamViewer 11MiB | | 0 N/A N/A 9003 G /usr/lib64/firefox/firefox 2MiB | +-----------------------------------------------------------------------------+

Now, any advice is appreciated.

Comments

3 comments

Chris Root

March 16, 2021 22:33
Hi,

Thanks for the info you provided. It helps to answer your question.

The first thing you're going to want to do is adjust your omnisci_server configuration file to account for additional gpu memory use by other processes. You can do this via the "res-gpu-mem" configuration flag (https://docs.omnisci.com/installation-and-configuration/config-parameters#configuration-parameters-for-omniscidb)

From your nvidia-smi output, it looks like there's approx 400MB of additional gpu mem being used by other processes. You're probably going to want to set "res-gpu-mem" to something higher than that to account for potentially other processes, and other residual gpu mem used by omnisci_server.

I'd say set that to something like 700MB - 1GB.

Also, you may not require the full 1GB of memory that the "CUDA_ERROR_UNKNOWN/CUDA_ERROR_NOT_SUPPORTED, possibly not enough gpu memory available for the requested buffer size of 1000000000 bytes on gpu 0." error refers to. The 1GB of memory requested here is controlled via the "render-mem-bytes" configuration parameter. You may want to try setting that lower depending on your use case if you're still crammed for memory.

You can try setting that to 500MB as well.

So, in summary, try adding:

res-gpu-mem = 700000000

and optionally: render-mem-bytes = 500000000

to your server config file and restart the server.
1

Comment actions Permalink
Gianfranco Campana

March 17, 2021 11:32
Thank you Christopher, those two params solved the issue for some additional editing, but then the problem raised again:

Chart Redraw Error warning_outline Exception: OutOfGpuMemoryError: Cuda error code=801, CUDA_ERROR_UNKNOWN/CUDA_ERROR_NOT_SUPPORTED, possibly not enough gpu memory available for the requested buffer size of 500000000 bytes on gpu 0.

I still have to complete this dashboard, and I'm a bit concerned about the memory lacking: I have built rich and complicated dashboards with a old version of MapD on the same hardware, I find strange that this simple one already suffers from memory lacking on Omnisci.
0

Comment actions Permalink
Candido Dessanti

March 17, 2021 13:22
Hi @Gianfranco_Campana,

In versions of omnisci prior to 4.0, the Choropleth map was a client-side object (You can get the same object starting the server in CPU mode), so it didn't take a single byte of GPU memory, while now it's rendered on the server, so we had to allocate a certain amount of memory to render this map.

probably the software allocated too much memory to render your map, but we are constantly improving the software so the tunable parameters,

Candido
0

Comment actions Permalink

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?