Datatype from a parquet column is not matched with HeavDB table
val locationIds = Some(List(LocationDashboardId(840))),
When this is written in a parquet file, I could see the schema below.
|-- locationIds: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- serializedId: array (nullable = true)
| | | |-- element: Integer (containsNull = true)
However, when this is being inserted to a heavydb table for the column with datatype INTEGER[] NOT NULL, the below error occurs.
Conversion from Parquet type "INT32" to HeavyDB type "INTEGER[]" is not allowed. Please use an appropriate column type. Parquet column: element, HeavyDB column: location_ids, Parquet file: /migration_csv/final/part-00000-680ff400-af6a-42d8-b471-fa654d0c844e-c000.snappy.parquet.
Please help me with the approach.
-
Hi Abdul Rahim Kamaludeen,
Could you share the parquet file you are trying to ingest or some sample rows?
In any case, could you run the \detect_parquet command from the heavysql command?
It'd return field names, datatypes, some sample data, and a DDL of the table.\detect_parquet /opt/opendata/flights/flights_none.parquet
[cut output]
CREATE TABLE your_table_name(flight_year SMALLINT, flight_mmonth SMALLINT, flight_dayofmonth SMALLINT, flight_dayofweek SMALLINT, deptime SMALLINT, crsdeptime SMALLINT, arrtime SMALLINT, crsarrtime SMALLINT, uniquecarrier TEXT ENCODING DICT, flightnum SMALLINT, tailnum TEXT ENCODING DICT, actualelapsedtime SMALLINT, crselapsedtime SMALLINT, airtime SMALLINT, arrdelay SMALLINT, depdelay SMALLINT, origin TEXT ENCODING DICT, dest TEXT ENCODING DICT, distance SMALLINT, taxiin SMALLINT, taxiout SMALLINT, cancelled SMALLINT, cancellationcode TEXT ENCODING DICT, diverted SMALLINT, carrierdelay SMALLINT, weatherdelay SMALLINT, nasdelay SMALLINT, securitydelay SMALLINT, lateaircraftdelay SMALLINT, dep_timestamp TIMESTAMP(3), arr_timestamp TIMESTAMP(3), carrier_name TEXT ENCODING DICT, plane_type TEXT ENCODING DICT, plane_manufacturer TEXT ENCODING DICT, plane_issue_date TEXT ENCODING DICT, plane_model TEXT ENCODING DICT, plane_status TEXT ENCODING DICT, plane_aircraft_type TEXT ENCODING DICT, plane_engine_type TEXT ENCODING DICT, plane_year SMALLINT, origin_name TEXT ENCODING DICT, origin_city TEXT ENCODING DICT, origin_state TEXT ENCODING DICT, origin_country TEXT ENCODING DICT, origin_lat FLOAT, origin_lon FLOAT, dest_name TEXT ENCODING DICT, dest_city TEXT ENCODING DICT, dest_state TEXT ENCODING DICT, dest_country TEXT ENCODING DICT, dest_lat FLOAT, dest_lon FLOAT);And post the reulteing DDLs?
Bests,
Candido
Please sign in to leave a comment.
Comments
1 comment