Skip to content

[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast #28758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raminqaf wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast #28758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8db28aa
[FLINK-37925][table] Reject CAST from a VARIANT to values that do not…
raminqaf Jul 16, 2026
2f826e6
[FLINK-37925][docs] Document VARIANT value encoding and PARSE_JSON ty…
raminqaf Jul 16, 2026
c3dada9
[FLINK-37925][core] Document VARIANT timestamp precision in the Varia…
raminqaf Jul 16, 2026
fd2676f
[FLINK-37925] Address feedbacks
raminqaf Jul 22, 2026
db85cdf
[FLINK-37925] Add strikt cast validation for VARCHAR and CHAR
raminqaf Jul 22, 2026
23e41c6
[FLINK-37925] Fail on casting Variant array, objects, bytes to String
raminqaf Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 51 additions & 6 deletions docs/content.zh/docs/sql/reference/data-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1086,7 +1086,7 @@ The type can be declared using the above combinations where `p1` is the number o
and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by default. If no `p2` is
specified, it is equal to `6` by default.

### Constructured Data Types
### Constructed Data Types

#### `ARRAY`

Expand Down Expand Up @@ -1507,19 +1507,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.

`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.

`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.

A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.
are stored with microsecond precision.


The `PARSE_JSON` function produces only the kinds that JSON syntax can express:

| JSON input | Stored `VARIANT` kind |
|-----------------------------------------------------|-------------------------------------------------|
| `null` | `NULL` |
| `true` / `false` | `BOOLEAN` |
| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
| Number exceeding 38 digits of precision or scale | `DOUBLE` |
| String | `STRING` |
| Array | array (elements encoded by the same rules) |
| Object | object (values encoded by the same rules) |

Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
conversions like Avro.

The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also include the 1e400 example here.

A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.

A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
targets accept any wider numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A

value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not go into the details of CHAR/VARCHAR at this line

Suggested change
handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
handled the same way. Casting a `VARIANT` to `STRING` extracts the scalar value, so a

stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
Comment on lines +1556 to +1557

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

people should know how they work already

Suggested change
padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
padding or truncation.


{{< hint info >}}
Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows. To narrow a
`VARIANT`, first cast it to a type that fits the stored value, then apply a
regular narrowing cast:

```sql
CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after overflow)
```
{{< /hint >}}
Comment on lines +1564 to +1567

@snuyanzin snuyanzin Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is difference here from non VARIANT case?

Comment on lines +1559 to +1567

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would vote for dropping this paragraph entirely. Or make it less prominent. A hint info is definitely too much. A sentence like Numeric overflows are prohibited. should be enough after: targets accept any wider numeric value, so PARSE_JSON('42')casts toINT, BIGINT, or DOUBLE. Numeric overflows are prohibited.


**Declaration**

Expand Down
59 changes: 52 additions & 7 deletions docs/content/docs/sql/reference/data-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1094,7 +1094,7 @@ The type can be declared using the above combinations where `p1` is the number o
and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by default. If no `p2` is
specified, it is equal to `6` by default.

### Constructured Data Types
### Constructed Data Types

#### `ARRAY`

Expand Down Expand Up @@ -1515,19 +1515,64 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.

`VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps,
or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity
or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity
and nesting are significant.

`VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas
without requiring upfront schema definition. For example, if a new field is added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.

A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer
such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of
the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns
`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`.
A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`,
`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`,
`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ`
are stored with microsecond precision and `DATE` as a day count. There is no `TIME` kind.

@davidradl davidradl Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why microsecond not nanoseconds?

@raminqaf raminqaf Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidradl This is how the implementation of the BinaryVariant defines it.

    @Override
    public Instant getInstant() throws VariantTypeException {
        checkType(Type.TIMESTAMP_LTZ, getType());
        return microsToInstant(BinaryVariantUtil.getLong(value, pos));
    }

The implementation is done based on the Encoding types defined in Parquet. Flink implements the types from 0 (NULL) to 16 (STRING). The rest of the types: 17 (Time Macro),18 Timestamp (Nano),19 TimestampNTZ (Nano), 20 (UUID) are not implemented in Flink yet.


@davidradl davidradl Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious why Date does not go to Flink date which would seem more natural

@raminqaf raminqaf Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidradl I think the implementation of DATE is also based on Parquet: https://parquet.apache.org/docs/file-format/types/logicaltypes/

DATE is used for a logical date type, without a time of day. It must annotate an int32 that stores the number of days from the Unix epoch, 1 January 1970.

    @Override
    public LocalDate getDate() throws VariantTypeException {
        checkType(Type.DATE, getType());
        return LocalDate.ofEpochDay(BinaryVariantUtil.getLong(value, pos));
    }

The `PARSE_JSON` function produces only the kinds that JSON syntax can express:

| JSON input | Stored `VARIANT` kind |
|-----------------------------------------------------|-------------------------------------------------|
| `null` | `NULL` |
| `true` / `false` | `BOOLEAN` |
| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` |
| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` |
| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` |
| Number exceeding 38 digits of precision or scale | `DOUBLE` |
| String | `STRING` |
| Array | array (elements encoded by the same rules) |
| Object | object (values encoded by the same rules) |

Because JSON has no literal for float, date, timestamp, or binary values, `PARSE_JSON` never
produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A `VARIANT` can still hold
those kinds when built by other producers, such as the programmatic `VariantBuilder` API or format
conversions like Avro.

The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`,
`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and `TRY_PARSE_JSON` returns `NULL`.
A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast
it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`.

A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric
targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`, `BIGINT`, or `DOUBLE`. A
value outside an integer or `DECIMAL` target's range fails `CAST` and returns `NULL` for
`TRY_CAST`. Other targets require the stored value to be of the matching kind, and a mismatch is
handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the scalar value, so a
stored string is returned unquoted (for example `foo`). A variant that holds an object, array, or
binary value is not castable to a string; use `JSON_STRING` for its JSON representation instead
(there a string stays quoted, for example `"foo"`). A bounded `CHAR(n)`/`VARCHAR(n)` target is length-checked strictly, with no
padding or truncation: `VARCHAR(n)` accepts up to `n` characters and `CHAR(n)` requires exactly `n`,
otherwise `CAST` fails and `TRY_CAST` returns `NULL`.

{{< hint info >}}
Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows. To narrow a
`VARIANT`, first cast it to a type that fits the stored value, then apply a
regular narrowing cast:

```sql
CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after overflow)
```
{{< /hint >}}

**Declaration**

Expand Down
4 changes: 4 additions & 0 deletions docs/data/sql_functions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1234,6 +1234,8 @@ variant:
parser will keep the last occurrence of all fields with the same key, otherwise when
`allowDuplicateKeys` is false it will throw an error. The default value of
`allowDuplicateKeys` is false.

See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds.
- sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys])
description: |
Try to parse a JSON string into a Variant if possible. If the JSON string is invalid, return
Expand All @@ -1243,6 +1245,8 @@ variant:
parser will keep the last occurrence of all fields with the same key, otherwise when
`allowDuplicateKeys` is false it will throw an error. The default value of
`allowDuplicateKeys` is false.

See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds.
- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
Expand Down
4 changes: 4 additions & 0 deletions docs/data/sql_functions_zh.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1320,6 +1320,8 @@ variant:
同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下，
allowDuplicateKeys 的值为 false。

See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds.

- sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys])
description: |
尽可能将 JSON 字符串解析为 Variant。如果 JSON 字符串无效，则返回 NULL。如果希望抛出错误而不是返回 NULL，
Expand All @@ -1329,6 +1331,8 @@ variant:
同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下，
allowDuplicateKeys 的值为 false。

See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds.

- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
Expand Down
15 changes: 9 additions & 6 deletions flink-core/src/main/java/org/apache/flink/types/variant/Variant.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ public interface Variant {
float getFloat() throws VariantTypeException;

/**
* Get the scalar value of variant as BigDecimal, if the variant type is {@link Type#DECIMAL}.
* Get the scalar value of variant as {@link BigDecimal}, if the variant type is {@link
* Type#DECIMAL}.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#DECIMAL}.
Expand All @@ -118,27 +119,29 @@ public interface Variant {
String getString() throws VariantTypeException;

/**
* Get the scalar value of variant as LocalDate, if the variant type is {@link Type#DATE}.
* Get the scalar value of variant as {@link LocalDate}, if the variant type is {@link
* Type#DATE}.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#DATE}.
*/
LocalDate getDate() throws VariantTypeException;

/**
* Get the scalar value of variant as LocalDateTime, if the variant type is {@link
* Type#TIMESTAMP}.
* Get the scalar value of variant as {@link LocalDateTime}, if the variant type is {@link
* Type#TIMESTAMP}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#TIMESTAMP}.
*/
LocalDateTime getDateTime() throws VariantTypeException;

/**
* Get the scalar value of variant as Instant, if the variant type is {@link Type#TIMESTAMP}.
* Get the scalar value of variant as {@link Instant}, if the variant type is {@link
* Type#TIMESTAMP_LTZ}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or is not {@link
* Type#TIMESTAMP}.
* Type#TIMESTAMP_LTZ}.
*/
Instant getInstant() throws VariantTypeException;

Expand Down
10 changes: 1 addition & 9 deletions ...-table/src/main/java/org/apache/flink/table/examples/java/basics/WordCountSQLExample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,7 @@ public static void main(String[] args) throws Exception {
// execute a Flink SQL job and print the result locally
tableEnv.executeSql(
// define the aggregation
"SELECT word, SUM(frequency) AS `count`\n"
// read from an artificial fixed-size table with rows and columns
+ "FROM (\n"
+ " VALUES ('Hello', 1), ('Ciao', 1), ('Hello', 2)\n"
+ ")\n"
// name the table and its columns
+ "AS WordTable(word, frequency)\n"
// group for aggregation
+ "GROUP BY word")
"SELECT PARSE_JSON('Infinity')")

@twalthr twalthr Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undo

.print();
}
}
10 changes: 0 additions & 10 deletions ...rc/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
import java.util.List;
import java.util.Optional;

import static org.apache.flink.table.types.logical.utils.LogicalTypeCasts.getUnsupportedCastHint;
import static org.apache.flink.table.types.logical.utils.LogicalTypeCasts.supportsExplicitCast;

/**
Expand Down Expand Up @@ -70,15 +69,6 @@ public Optional<List<DataType>> inferInputTypes(
return Optional.of(argumentDataTypes);
}
if (!supportsExplicitCast(fromType, toType)) {
final Optional<String> hint = getUnsupportedCastHint(fromType, toType);
if (hint.isPresent()) {
return callContext.fail(
throwOnFailure,
"Unsupported cast from '%s' to '%s'. %s",
fromType,
toType,
hint.get());
}
return callContext.fail(
throwOnFailure, "Unsupported cast from '%s' to '%s'.", fromType, toType);
}
Expand Down
Loading