Using HeavyAI connector in NodeJS application
I want to use the @heavyai connector in a NodeJS application. I've tried several variations of import statements, including:
import DbCon from "@heavyai/connector"
None of the variations I tried work, and I can only find documentation for running JavaScript in the browser.
How do I import the connector into a NodeJS app?
Is there documentation specific to NodeJS?
-
Hello Lee Jenkins
We have provided some samples on how to use the connector without relying on the browser.
You can access them via this link: https://github.com/heavyai/heavyai-connector/blob/master/test/integration.spec.js
Please let me know if these examples meet your requirements.
Best regards,
Candido -
Hi Lee,
Our Front-End Engineer re-tested version 7.0 of the connector using Node.js to ensure that it works in both successful and failed scenarios. They found that everything is functioning as intended.
The code used:const Connector = require("@heavyai/connector/dist/node-connector.js").DbCon; const hostname = process.env.HOSTNAME || "kali.mapd.com" const protocol = process.env.PROTOCOL || "https" const port = process.env.PORT || "10043" const database = process.env.DATABASE || "mapd" const username = process.env.USER_NAME || "admin" const password = process.env.PASSWORD || "HyperInteractive" const config = { hostname, protocol, port, database, username, password, } const connector = new Connector() .protocol(config.protocol) .host(config.hostname) .port(config.port) .dbName(config.database) .user(config.username) .password(config.password); console.log(`connect to heavydb...`); connector.connect((err, session) => { if (err) { console.error(err) throw err } console.log(`connect succeeded! ${session.sessionId()}`); });This test rules out any problems with the connector, so it's possible that we are facing an infrastructure issue on your side. For reference, here is the output of the environment they used for testing.
A summary of component version with a successful connection.
Using a not-existent host, so a failure.
Do this helps?
Candido -
Hi Candido.
Thanks for posting that example. I tried following the steps in the example, but my example nodejs program still cannot connect to the database. My tests are described below. Could you take a look at what I am doing, or forward this to someone who might be able to help? Whatever I'm missing, it is eluding me.
Regards,
Lee Jenkins-----------------------------------------------------------
Here is a summary of what I have tried:
- created an env file with host/port/user/etc. variables.
- created a shell script (below) to execute a query, using the CLI heavysql to connect with env vars
- created a nodejs script (below) to execute the same query, using the heavyai-connector example code and the env vars
The shell script that calls heavysql can connect and execute the query very quickly. The nodejs script attempts to connect, but the callback function is never executed. I expected the callback function to be called with either an error or a session.
As an experiment, I also tried calling `connector.connectAsync()` instead of `connector.connect()`. The result was effectively the same: the promise was never resolved or rejected. The script just hung at the call to `connectAsync()`.
For negative test cases, I changed to port number in the env file and ran both scripts again. Both scripts immediately reported connection errors. So that tells me that the nodejs connector is trying to connect. But unlike the command-line heavysql, the nodejs script is hanging during the connect.
Here is a log of the tests:
$ node --versionv18.14.0$ !grepgrep version node_modules/@heavyai/connector/package.json"version": "7.0.0",$ heavysql --versionHeavySQL Version: 6.4.0-20230201-87f6fa37b4$ time ./heavysql-testDB_PROTOCOL is 'mapd'connect with: heavysql myDatabase -s remote-host --port 6274 -u myname -p mypasswdUser myname connected to database myDatabaseEXPR$01071552723User myname disconnected from database myDatabase
real 0m0.381suser 0m0.003ssys 0m0.004s$ time node test-me.jsconfig hostname = 'remote-host'config protocol = 'mapd'config port_num = '6274'config database = 'myDatabase'config username = 'myname'config password = 'mypasswd'connect to heavydb...^C
real 0m9.997suser 0m0.255ssys 0m0.040s$Here is the nodejs script test-me.js:
const Connector = require("@heavyai/connector/dist/node-connector.js").DbCon;
const config = {hostname: process.env.DB_HOSTNAME,protocol: process.env.DB_PROTOCOL,port_num: process.env.DB_PORT_NUM,database: process.env.DB_DATABASE,username: process.env.DB_USERNAME,password: process.env.DB_PASSWORD,};
Object.keys(config).forEach(propertyName => {const propertyValue = config[propertyName];if (typeof propertyValue !== 'string') exitError(`missing heavydb config property ${propertyName}`);console.log(`config ${propertyName} = '${propertyValue}'`);});
const connector = new Connector().protocol(config.protocol).host(config.hostname).port(config.port_num).dbName(config.database).user(config.username).password(config.password);
console.log(`connect to heavydb...`);
const useCallbackInterface = false;
if (useCallbackInterface) {connector.connect((err, session) => {if (err) exitError(err);console.log(`connect succeeded!`);executeQuery(session);});}else {connector.connectAsync().then(session => {console.log(`connect succeeded!`);executeQuery(session);}).catch(err => {console.log(`connection error:`);exitError(err);});}
function executeQuery(session) {const query = `select count(*) from telemetry_test_alpha;`console.log(`send query to heavydb: ${query}`);session.query(query, (err, data) => {if (err) exitError(err);console.log(`${data.toString()}\n`);});}
function exitError(err) {console.error(err);process.exit(1001);}And here is the shell script heavysql-test:
#!/bin/bash
QUERY="select count(*) from telemetry_test_alpha;"
PROTOCOL=
echo "DB_PROTOCOL is '$DB_PROTOCOL'"
if [[ "$DB_PROTOCOL" = "https" || "$DB_PROTOCOL" = "http" ]]; thenPROTOCOL="--$DB_PROTOCOL"fi
echo "connect with: heavysql $DB_DATABASE $PROTOCOL -s $DB_HOSTNAME --port $DB_PORT_NUM -u $DB_USERNAME -p $DB_PASSWORD"
echo "$QUERY" | heavysql $DB_DATABASE $PROTOCOL -s $DB_HOSTNAME --port $DB_PORT_NUM -u $DB_USERNAME -p $DB_PASSWORD -
Yes, that is true. The NodeJS program hangs when trying to connect and all the supplied connection information is correct. In an attempt to troubleshoot the program, I also tested it with incorrect connection information. If I supply incorrect information, such as the wrong port number for the HeavyDB service, then the NodeJS app immediately reports an error. In this case, the program does not hang.
FYI The javascript connector for HeavyAI uses a "callback" interface, where HeavyAI is supposed to call the callback function if the connection succeeds *or* fails. But in my tests, HeavyAI only calls the callback when the connection fails.
Hope that helps.
-
I have narrowed down the issue. The program was using PROTOCOL=mapd in the connection configuration.
When running the code your front-end engineer supplied, it would hang if it was configured with PROTOCOL=mapd, but it would connect successfully if it was configured with PROTOCOL=https.
Next I ran my nodejs program with PROTOCOL=https, and it, too, was able to connect successfully.
Of course, these protocols answer on different ports, so the configuration change also reflected the correct port numbers.
Finally, I ran the `heavysql` command with both configurations, mapd and https. The `heavysql` command connected successfully with both protocols.
My conclusion is that the javascript connector is not working correctly with the mapd protocol.
Now that I know, I can simply use the https protocol.
-
There isn't a 'mapd' protocol (probably we should review the documentation) . Port 6274 uses a TCP binary protocol, while port 6276, which is a recent addition, utilizes a binary HTTP protocol. Neither of these ports is compatible with the JS connector, but this isn't an issue since the JS connector is designed for web applications.
Anyway, I'm glad to hear that it's now working for you.
Please sign in to leave a comment.
Comments
9 comments