Data Types
Data Types#
DataFusion uses Arrow, and thus the Arrow type system, for query execution. The SQL types from sqlparser-rs are mapped to Arrow data types according to the following table. This mapping occurs when defining the schema in a CREATE EXTERNAL TABLE command or when performing a SQL CAST operation.
For background on extension types and custom metadata, see the Implementing User Defined Types and Custom Metadata in DataFusion blog.
You can see the corresponding Arrow type for any SQL expression using the arrow_typeof function. For example:
select arrow_typeof(interval '1 month');
+---------------------------------------------------------------------+
| arrow_typeof(IntervalMonthDayNano("79228162514264337593543950336")) |
+---------------------------------------------------------------------+
| Interval(MonthDayNano) |
+---------------------------------------------------------------------+You can cast a SQL expression to a specific Arrow type using the arrow_cast function For example, to cast the output of now() to a Timestamp with second precision:
select arrow_cast(now(), 'Timestamp(s)') as "now()";
+---------------------+
| now() |
+---------------------+
| 2025-10-24T20:02:45 |
+---------------------+The older syntax still works as well:
select arrow_cast(now(), 'Timestamp(Second, None)') as "now()";
+---------------------+
| now() |
+---------------------+
| 2023-03-03T17:19:21 |
+---------------------+Character Types#
| SQL DataType | Arrow DataType |
|---|---|
CHAR | Utf8View |
VARCHAR | Utf8View |
TEXT | Utf8View |
STRING | Utf8View |
CHARACTER VARYING | Utf8View |
CHAR VARYING | Utf8View |
NVARCHAR | Utf8View |
By default, string types are mapped to Utf8View. This can be configured using the datafusion.sql_parser.map_string_types_to_utf8view setting. When set to false, string types are mapped to Utf8 instead.
Note that CHAR, VARCHAR, STRING, CHARACTER VARYING, CHAR VARYING, and NVARCHAR can optionally receive the maximum length (e.g., VARCHAR(n)). However, this limit is ignored. To raise an error when the length is passed, set datafusion.sql_parser.support_varchar_with_length to false.
Numeric Types#
| SQL DataType | Arrow DataType |
|---|---|
TINYINT | Int8 |
SMALLINT or INT2 | Int16 |
INT or INTEGER or INT4 | Int32 |
BIGINT or INT8 | Int64 |
TINYINT UNSIGNED or UTINYINT | UInt8 |
SMALLINT UNSIGNED or INT2 UNSIGNED or USMALLINT | UInt16 |
INT UNSIGNED or INTEGER UNSIGNED or INT4 UNSIGNED | UInt32 |
BIGINT UNSIGNED or INT8 UNSIGNED or UBIGINT | UInt64 |
FLOAT or FLOAT4 or REAL | Float32 |
DOUBLE or DOUBLE PRECISION or FLOAT8 | Float64 |
DECIMAL(precision, scale) or NUMERIC(precision, scale) where precision ≤ 38 | Decimal128(precision, scale) |
DECIMAL(precision, scale) or NUMERIC(precision, scale) where precision > 38 | Decimal256(precision, scale) |
The maximum supported precision for DECIMAL types is 76.
Date/Time Types#
| SQL DataType | Arrow DataType |
|---|---|
DATE | Date32 |
TIME | Time64(Nanosecond) |
TIMESTAMP | Timestamp(Nanosecond, None) |
INTERVAL | Interval(IntervalMonthDayNano) |
Boolean Types#
| SQL DataType | Arrow DataType |
|---|---|
BOOLEAN or BOOL | Boolean |
Binary Types#
| SQL DataType | Arrow DataType |
|---|---|
BYTEA | Binary |
BLOB | Binary |
BINARY | Binary |
VARBINARY | Binary |
BYTES | Binary |
You can create binary literals using a hex string literal such as X'1234' to create a Binary value of two bytes, 0x12 and 0x34.
Nested Types#
| SQL DataType | Arrow DataType |
|---|---|
ARRAY<T> or T[] | List(T) |
T[n] | FixedSizeList(n x T) |
MAP(k, v) | Map(k, v) |
STRUCT<[c0] T0[, ..., [cn] Tn]> | Struct(c0: T0[, ..., cn: Tn]) |
While columns defined with STRUCT require all rows to have the same schema, MAP imposes no such restriction.
Also note that STRUCT can be declared without defining the field names. In that case, they will be set to c0, c1, …, cn.
Unsupported SQL Types#
| SQL Data Type | Arrow DataType |
|---|---|
UUID | Not yet supported |
CLOB | Not yet supported |
REGCLASS | Not yet supported |
CUSTOM | Not yet supported |
ENUM | Not yet supported |
SET | Not yet supported |
DATETIME | Not yet supported |
评论
登录后参与评论
KnowForge