feat(format): add generic metadata API - #4694
Conversation
|
I'm thinking we don't run-length-encode by default to simplify things further. |
f4b9198 to
e222af1
Compare
e222af1 to
1a39168
Compare
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Exciting! I've left a few comments and questions resulting from a first, casual review of the code.
| /// |--------------------------|------------------------------|----------| | ||
| /// | collection_name | utf8 not null | | | ||
| /// | collection_description | utf8 | | | ||
| /// | collection_schema | extension<arrow.schema_json> | | |
There was a problem hiding this comment.
The proposal for this type seems to have stalled :(.
There was a problem hiding this comment.
I'm talking to Kent, hopefully we can keep it going. Otherwise, I may propose an arrow.schema_ipc (even if that's a bit ugly)
|
|
||
| /// \brief The "meta" collection returns the available metadata collections. | ||
| /// | ||
| /// | Field Name | Field Type | Comments | |
There was a problem hiding this comment.
Do we need to say that the field types can be logical vs physical e.g. REE- or dictionary-encoded? Or is that always implicit? EDIT: I just saw that the consumer can explicitly ask for this kind of encoding if that's what they want. This doesn't necessarily imply that the producer is otherwise constrained to provide e.g. utf8 instead of utf8_view so I still think it would be value to express that explicitly if it's the case.
There was a problem hiding this comment.
I think we should require drivers to stick to the declared type unless requested, I can clarify this
| /// \param[out] error An optional location to return an error message if necessary. | ||
| /// | ||
| /// \return ADBC_STATUS_OK on success or an appropriate error code. | ||
| ADBC_EXPORT |
There was a problem hiding this comment.
Should these other changes/fixes be moved to a separate PR?
| /// | ||
| /// Drivers may add more fields at the end of standard schemas to reflect | ||
| /// vendor-specific metadata. Applications must access these using an offset | ||
| /// from the end of the schema and cannot assume that the index of the field |
There was a problem hiding this comment.
Do we want to generally discourage access by ordinal?
| /// - ADBC_METADATA_FILTER_PATTERN_COLUMN | ||
| #define ADBC_METADATA_COLLECTION_ROUTINE_COLUMNS "routine_columns" | ||
|
|
||
| /// \brief The "namespaces" collection returns a level of namespaces defined |
There was a problem hiding this comment.
These don't seem to interact much with the other metadata, or at least I'm having trouble seeing how they would. Do you have an example of a scenario where these matter, and how they would impact e.g. the listing of tables?
There was a problem hiding this comment.
Ah, I can take this out...I was trying to explore how things might work for databases that don't follow the catalog-schema-table hierarchy, especially now that Iceberg/Iceberg REST Catalog has introduced arbitrarily-nested namespaces into systems. But as mentioned I think we would basically have to duplicate all the metadata collections to support that and it's not worth trying to make them both fit into the same collections.
2 participants
Related:
Closes #4400.