Scalar Functions Support Status

Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 247 functions and partially supports 28 functions.

Gluten also fully supports 2 additional functions introduced in Spark 4.0.

Array Functions

Bitwise Functions

Collection Functions

Conditional Functions

Conversion Functions

Csv Functions

Date and Timestamp Functions

Hash Functions

JSON Functions

Spark Functions Spark Expressions Status Restrictions

from_json JsonToStructs PS from_json with ‘spark.sql.caseSensitive = true’ is not supported in Velox
from_json with ‘spark.sql.json.enablePartialResults = false’ is not supported in Velox
from_json with column corrupt record is not supported in Velox
from_json with duplicate keys is not supported in Velox
from_json with options is not supported in Velox

get_json_object GetJsonObject S  

json_array_length LengthOfJsonArray S  

json_object_keys JsonObjectKeys S  

json_tuple JsonTuple S  

schema_of_json SchemaOfJson    

to_json StructsToJson PS When ‘spark.sql.caseSensitive = false’, to_json produces unexpected result for struct field with uppercase name
to_json with options is not supported in Velox

Lambda Functions

Map Functions

Mathematical Functions

Misc Functions

Predicate Functions

String Functions

Spark Functions Spark Expressions Status Restrictions

ascii Ascii S  

base64 Base64 PS base64 with chunkBase64String disabled is not supported

bit_length BitLength S  

btrim StringTrimBoth S  

char Chr S  

char_length Length S  

character_length Length S  

chr Chr S  

concat_ws ConcatWs S  

contains ContainsExpressionBuilder PS BinaryType unsupported

decode Decode    

elt Elt    

encode Encode    

endswith EndsWithExpressionBuilder PS BinaryType unsupported

find_in_set FindInSet S  

format_number FormatNumber PS format_number only supports tinyint, smallint, integer, bigint, float and double input; DecimalType input is not supported in Velox
format_number with a string format argument (e.g. ‘#,###.##’) is not supported in Velox; only an integer number of decimal places is supported

format_string FormatString    

initcap InitCap S  

instr StringInstr S  

lcase Lower S  

left Left S  

len Length S  

length Length S  

levenshtein Levenshtein S  

locate StringLocate S  

lower Lower S  

lpad LPadExpressionBuilder PS BinaryType unsupported

ltrim StringTrimLeft S  

luhn_check Luhncheck S  

mask MaskExpressionBuilder S  

octet_length OctetLength    

overlay Overlay S  

position StringLocate S  

printf FormatString    

regexp_count RegExpCount    

regexp_extract RegExpExtract PS Lookaround unsupported

regexp_extract_all RegExpExtractAll PS Lookaround unsupported

regexp_instr RegExpInStr PS Lookaround unsupported, group index ignored

regexp_replace RegExpReplace PS Lookaround unsupported

regexp_substr RegExpSubStr    

repeat StringRepeat S  

replace StringReplace S  

right Right S  

rpad RPadExpressionBuilder PS BinaryType unsupported

rtrim StringTrimRight S  

sentences Sentences    

soundex SoundEx S  

space StringSpace    

split StringSplit S  

split_part SplitPart S  

startswith StartsWithExpressionBuilder PS BinaryType unsupported

substr Substring PS  

substring Substring PS  

substring_index SubstringIndex S  

to_binary ToBinary    

to_char ToCharacter    

to_number ToNumber    

to_varchar ToCharacter    

translate StringTranslate S  

trim StringTrim S  

try_to_binary TryToBinary    

try_to_number TryToNumber    

ucase Upper S  

unbase64 UnBase64 PS unbase64 with failOnError is not supported

upper Upper S  

Struct Functions

URL Functions

XML Functions