feat: support reading Lance tables on Hadoop-family filesystems (tbdsfs/hdfs) - #73
Merged
fightBoxing merged 4 commits intoAug 27, 2026
Conversation
added 3 commits
August 27, 2026 12:49
…fs/hdfs)
Lance's Rust object_store natively supports only s3/gs/az/cos/oss/hf/file
schemes. Hadoop-family URIs (tbdsfs://, hdfs://, etc.) previously failed
with 'No object store provider found for scheme'.
This change introduces a transparent Hadoop-FileSystem-based fallback:
* New util LanceHadoopPathResolver:
- Detect non-native schemes; pass native ones through.
- Normalize URIs like 'scheme:/x' back to 'scheme://x' (some upstream
code paths collapse the double slash and lose the authority).
- Build Hadoop Configuration, injecting values from system properties
(lance.hadoop.*) and env vars (LANCE_HADOOP_*) so TaskManagers can
receive tbdsfs.meta etc. without core-site.xml surgery.
- Apply tbdsfs-specific defaults: auto-derive tbdsfs.name from URI
authority, fill fs.tbdsfs.impl if missing.
- Download the whole dataset directory into a local cache
($java.io.tmpdir/lance-hadoop-cache by default) and return the
'file://' path so lance-core can read it natively.
- Serialize concurrent resolves for the same URI; skip re-download
when cache already ready (detected via _versions/ subdir).
* LanceOpener.open now delegates path resolution to the resolver before
every Dataset.open call.
* pom.xml (lance-flink-1.20): add hadoop-common as provided dependency.
* Unit tests cover scheme detection, URI normalization, native-scheme
passthrough, Hadoop-based download via a LocalFileSystem shim, and
cache reuse behaviour.
lance-core's Rust JNI hard-codes Arrow C Data Interface class names (org.apache.arrow.c.jni.PrivateData, org.apache.arrow.c.Data, org.apache.arrow.c.ArrowArray) and resolves them via FindClass() at runtime. The maven-shade relocation of org.apache.arrow therefore breaks the JNI bridge, failing with: Could not find class Lorg/apache/arrow/c/jni/PrivateData; Remove the relocation so arrow keeps its original package names.
…st manifest Two fixes needed to read Lance tables on tbdsfs/hdfs in YARN TaskManagers: - Propagate flink.hadoop.* and WITH-clause hadoop.* options (e.g. tbdsfs.meta, fs.tbdsfs.impl) into the Hadoop Configuration used by LanceHadoopPathResolver, bypassing nodes whose core-site.xml is missing tbdsfs config (previously failed with 'invalid uri: /internal'). - Skip the unreadable latest-version sentinel manifest (u64::MAX-1 = 18446744073709551614.manifest), which some tbdsfs backends store corrupt and return input/output error for; Lance derives latest by scanning _versions/ so it can be safely skipped.
LanceHadoopPathResolver (in shared src/main/java) imports org.apache.hadoop.*. The hadoop-common dependency was only added to lance-flink-1.20/pom.xml, so the 1.18 and 1.19 modules failed to compile with 'package org.apache.hadoop.conf does not exist'. Add the same provided-scope hadoop-common dependency to both modules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for reading Lance tables stored on Hadoop-family filesystems (tbdsfs, hdfs), built on top of the existing time-travel layer. This also fixes three bugs that blocked tbdsfs reads in YARN.
Changes
feat: Hadoop-family filesystem fallback (
929672a)LanceHadoopPathResolverdetects non-native schemes (tbdsfs/hdfs) and downloads the dataset to a local cache via the HadoopFileSystembefore handing it to lance-core.LanceOpenerroutes dataset opening through the resolver so it composes withread.version/read.as-of-timestamp.LanceHadoopPathResolverTest.fix: stop shading org.apache.arrow (
0c52ed3)org.apache.arrowrelocation that broke lance-core's Rust JNI bridge.fix: propagate hadoop config + tolerate broken latest manifest (
1830644)flink.hadoop.*config and WITH-clausehadoop.*options (e.g.tbdsfs.meta,fs.tbdsfs.impl) into the HadoopConfigurationused for path resolution, so the table DDL can carry tbdsfs config directly.18446744073709551614.manifest), which some tbdsfs backends store corrupt.Test plan
Verified end-to-end in Flink 1.20 on a TBDS cluster:
SELECT COUNT(*)returned the expected 11 rows.ARRAY<FLOAT>vector column both read correctly (CARDINALITY(vector) = 512).Closes #70
Closes #71
Closes #72