Curl Error on make

February 24, 2021 20:00

I just downloaded the source code from github and ran the mapd-deps-prebuilt.sh to download the dependencies folder. As far as I can tell I have downloaded all the dependencies.

my cmake command runs fine

cmake -DCMAKE_BUILD_TYPE=debug ..

but when i run

make -j 4

I get the following errors. Can anybody shed some light on why this might be the case or how I can debug this issue. fatal: not a git repository (or any of the parent directories): .git -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_getinfo@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_global_init@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_perform@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_slist_free_all@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_strerror@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_easy_setopt@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_init@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_slist_append@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_cleanup@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_version_info@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_reset@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference tocurl_global_cleanup@CURL_OPENSSL_4' collect2: error: ld returned 1 exit status make[2]: [Tests/CMakeFiles/ArrowIpcIntegrationTest.dir/build.make:137: Tests/ArrowIpcIntegrationTest] Error 1 make[1]: [CMakeFiles/Makefile2:2796: Tests/CMakeFiles/ArrowIpcIntegrationTest.dir/all] Error

Comments

5 comments

Candido Dessanti

February 24, 2021 21:44
Which environment are you building the source in? I got a similar issue (mismatch of curl version) on Ubuntu 16.04 but it's unlikely that is the case.

If the problem is circumscribed to arrow tests only, you can use the enable_test switch in CMake to skip the generation of unit tests for the database
0

Comment actions Permalink
arif saeed

February 25, 2021 06:04
I am on Ubuntu 20.04.
I will try skipping unit tests as a temporary workaround.

how did you identify the curl version mismatch? when I type curl --version I get "curl 7.71.1" but I'm not sure if the code is looking at the /usr/local/mapd-deps for curl or another previously installed version.

Do you have any hints on how I can start debugging this issue? i.e. what part of the codebase contains the import of the library or tests

Thank you for replying to my question.
0

Comment actions Permalink
Candido Dessanti

February 25, 2021 10:47
Hi @asaeed,

The problem is happening in the linking phase, so to get the command that's failing, I suggest running the make with just 1 worker using the verbose switch turned to 1. something like make -j 1 VERBOSE=1

I talked about a mismatch because of the undefined reference errors, so the libarrow was compiled using a different libcurl-openssl, or the library isn't found at all. You can check which library the libarrow.so is looking for using the ldd command.

in the case of my ubuntu 18.04 installation

candido@zion-legion:/usr/local/mapd-deps/lib$ ldd libarrow.so | grep curl libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.s/usr/lib/x86_64-linux-gnu/libcurl.so.4o.4 (0x00007fd30877b000) candido@zion-legion:/usr/local/mapd-deps/lib$ ldd libarrow.so | grep ssl libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007ff68825a000)

checking the libcurl with nm command
```
candido@zion-legion:/usr/local/mapd-deps/lib$ nm -D /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep curl_easy
000000000002c450 T curl_easy_cleanup
000000000002c540 T curl_easy_duphandle
0000000000024b10 T curl_easy_escape
000000000002c4e0 T curl_easy_getinfo
000000000002c150 T curl_easy_init
000000000002c960 T curl_easy_pause
000000000002c1c0 T curl_easy_perform
000000000002cae0 T curl_easy_recv
000000000002c860 T curl_easy_reset
000000000002cb70 T curl_easy_send
0000000000057c00 T curl_easy_setopt
0000000000038330 T curl_easy_strerror
0000000000024df0 T curl_easy_unescape
```
Anyway I reproduced the error; it looks you are missing the libcurl.so.4 (or it's not in the path expected by libarrow.so)

you should check and eventually install libcurl4-openssl-dev. For reference in my system the version installed is 7.58.0-2ubuntu3.12

please let me know if the problem is solved

Regards, Candido
0

Comment actions Permalink
arif saeed

February 26, 2021 11:35
Thank you for your detailed instructions i really appreciated it. The problem is unfortunately still unresolved.

I tried to install libcurl4-openssl-dev and got the following message libcurl4-openssl-dev is already the newest version (7.68.0-1ubuntu2.4).

I started again and ran CMake and whilst it did complete generation, i got lots of the following errors, now sure if i should worry about them
CMake Warning at CMakeLists.txt:670 (add_executable): Cannot generate a safe runtime search path for target initdb because there is a cycle in the constraint graph:

dir 0 is [/home/arif/anaconda3/lib] dir 1 must precede it due to runtime library [libtbb.so] dir 1 is [/usr/local/mapd-deps/lib] dir 0 must precede it due to runtime library [libblosc.so.1]

Some of these libraries may not be found correctly.
I then ran make with the verbose setting, and the code failed with the following error

/usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4'

So i ran the the ld command on both libgdal and arrow libraries

ldd libgdal.so | grep curl ldd libarrow.so | grep curl

they gave the following locations respecttively.

libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f0ebcdad000) libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007ff142471000)

I then ran the following command

nm -D /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep curl_easy

I got the following

0000000000030e30 T curl_easy_cleanup 0000000000030f30 T curl_easy_duphandle 0000000000029140 T curl_easy_escape 0000000000030ed0 T curl_easy_getinfo 0000000000030b90 T curl_easy_init 00000000000313e0 T curl_easy_pause 0000000000030c00 T curl_easy_perform 00000000000315a0 T curl_easy_recv 00000000000312c0 T curl_easy_reset 0000000000031650 T curl_easy_send 000000000005d760 T curl_easy_setopt 000000000003d0f0 T curl_easy_strerror 0000000000029460 T curl_easy_unescape 0000000000031730 T curl_easy_upkeep

below is the command that was running immediately before it failed. I'm not sure if this is relevant or not but there is no mention of libcurl.so.4 in the list below, and the only mention of curl is /home/arif/anaconda3/lib/libcurl.so Could this be the mismatch mentioned could exist? If it is, I'm not sure what I need to do to fix the problem.

[ 60%] Linking CXX executable bin/initdb /usr/local/mapd-deps/bin/cmake -E cmake_link_script CMakeFiles/initdb.dir/link.txt --verbose=1 /usr/bin/c++ -Wall -Wno-unused-local-typedefs -fdiagnostics-color=auto -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -pthread -g -Wl,--export-dynamic -rdynamic CMakeFiles/initdb.dir/initdb.cpp.o -o bin/initdb -L/home/arif/anaconda3/lib -L/usr/local/mapd-deps/lib -Wl,-rpath,/usr/local/mapd-deps/lib:/home/arif/anaconda3/lib: libmapd_thrift.a DataMgr/libDataMgr.a OSDependent/Unix/libOSDependent.a Shared/libShared.a Catalog/libCatalog.a SqliteConnector/libSqliteConnector.a MigrationMgr/libMigrationMgr.a TableArchiver/libTableArchiver.a Parser/libParser.a Analyzer/libAnalyzer.a ImportExport/libImportExport.a QueryRunner/libQueryRunner.a QueryEngine/libQueryEngine.a ThriftHandler/libQueryState.a LockMgr/libLockMgr.a DataMgr/libDataMgr.a Fragmenter/libFragmenter.a Logger/libLogger.a Geospatial/libGeospatial.a Calcite/libCalcite.a /usr/local/mapd-deps/lib/libarrow_cuda.so /usr/local/mapd-deps/lib/libaws-cpp-sdk-s3.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-identity-management.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-cognito-identity.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-core.a /usr/local/mapd-deps/lib/libaws-c-event-stream.a /usr/local/mapd-deps/lib/libaws-c-common.a /usr/local/mapd-deps/lib/libaws-checksums.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-sts.a /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /usr/local/mapd-deps/lib/libarrow.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /home/arif/anaconda3/lib/libsnappy.so -ldl /usr/local/mapd-deps/lib/libfolly.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /usr/local/mapd-deps/lib/libfmt.a /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so -ldl /usr/lib/x86_64-linux-gnu/libunwind.so /usr/lib/x86_64-linux-gnu/libglog.so /usr/lib/x86_64-linux-gnu/libgflags.so -ltbb /usr/lib/x86_64-linux-gnu/libboost_log.so /usr/lib/x86_64-linux-gnu/libboost_log_setup.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_thread.so /usr/lib/x86_64-linux-gnu/libboost_timer.so /usr/lib/x86_64-linux-gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so -ldl /usr/local/cuda/lib64/libcudart_static.a -lpthread -ldl /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libcuda.so /home/arif/anaconda3/lib/libz.so TableArchiver/libTableArchiver.a Parser/libParser.a TableArchiver/libTableArchiver.a Parser/libParser.a LockMgr/libLockMgr.a Parser/libParserGenerated.a ImportExport/libImportExport.a libmapd_thrift.a DataMgr/libDataMgr.a CudaMgr/libCudaMgr.a /usr/local/cuda/lib64/libcudart_static.a -lpthread /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libcuda.so /home/arif/anaconda3/lib/libarchive.so /home/arif/anaconda3/lib/libbz2.so /home/arif/anaconda3/lib/liblzma.so /usr/local/mapd-deps/lib/libparquet.so Catalog/libCatalog.a MigrationMgr/libMigrationMgr.a Analyzer/libAnalyzer.a QueryEngine/libQueryEngine.a Calcite/libCalcite.a Catalog/libCatalog.a MigrationMgr/libMigrationMgr.a Analyzer/libAnalyzer.a QueryEngine/libQueryEngine.a Calcite/libCalcite.a SqliteConnector/libSqliteConnector.a Fragmenter/libFragmenter.a ThirdParty/bcrypt/libbcrypt.a /usr/local/mapd-deps/lib/libarrow_cuda.so /usr/local/mapd-deps/lib/libaws-cpp-sdk-s3.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-identity-management.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-cognito-identity.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-core.a /usr/local/mapd-deps/lib/libaws-c-event-stream.a /usr/local/mapd-deps/lib/libaws-c-common.a /usr/local/mapd-deps/lib/libaws-checksums.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-sts.a /usr/local/mapd-deps/lib/libarrow.so /home/arif/anaconda3/lib/libsnappy.so StringDictionary/libStringDictionary.a /usr/local/mapd-deps/lib/libfolly.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /usr/local/mapd-deps/lib/libfmt.a /usr/lib/x86_64-linux-gnu/libunwind.so /usr/lib/x86_64-linux-gnu/libglog.so /usr/lib/x86_64-linux-gnu/libgflags.so -ltbb Shared/libThriftClient.a ThirdParty/sqlite3/libsqlite3.a -ldl Utils/libUtils.a Shared/libShared.a Logger/libLogger.a /home/arif/anaconda3/lib/libblosc.so /usr/local/mapd-deps/lib/libclang-cpp.so /usr/local/mapd-deps/lib/libLLVM.so Calcite/libcalciteserver_thrift.a /usr/local/mapd-deps/lib/libthrift.so ThriftHandler/libQueryState.a /usr/lib/x86_64-linux-gnu/libboost_log.so /usr/lib/x86_64-linux-gnu/libboost_log_setup.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_thread.so /usr/lib/x86_64-linux-gnu/libboost_timer.so /usr/lib/x86_64-linux-gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so OSDependent/Unix/libOSDependent.a /usr/local/mapd-deps/lib/libgdal.so /home/arif/anaconda3/lib/libpng.so /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /home/arif/anaconda3/lib/libz.so /usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4'
0

Comment actions Permalink
Candido Dessanti

February 26, 2021 13:00
hi @asaeed,

I'm sorry for the not so fast reply because someone marked our message as SPAM for whatever reason.

without the generation of TEST, the error message has changed and it looks like the paths of miniconda eviroment are interfering with the mapd build environment; so probably is better to create an ad-hoc user for the build environment (or at least unset miniconda path before run cmake)

if you look at the verbose compilation command some of libraries of the anconda environment are being used, so the errors

e.g. gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so OSDependent/Unix/libOSDependent.a /usr/local/mapd-deps/lib/libgdal.so /home/arif/anaconda3/lib/libpng.so /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /home/arif/anaconda3/lib/libz.so /usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4’

Hopes this helps, Candido
0

Comment actions Permalink

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?