Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ public void initialize(String name, CaseInsensitiveStringMap options) {
}

// Initialize the namespace with proper configuration
Map<String, String> namespaceOptions = new HashMap<>(options);
Map<String, String> namespaceOptions = new HashMap<>(options.asCaseSensitiveMap());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on board with the direction: namespaceOptions and storageOptions at line 208 should have been the same key shape all along. What I want to check is that this drops the normalization wholesale instead of keeping it for the keys that actually need case preserved. Two things. First, the backend matches keys exactly (properties.get("root") and k.strip_prefix("storage.") in dir.rs, strip_prefix("headers.") in rest.rs). So ROOT=/tmp now fails with Missing required property 'root', wrapped in Failed to construct namespace impl …, while Headers.x-api-key, credential_vendor.Enabled and Storage.region are dropped with no error at all. Case in the suffix alone is still fine: storage.Region works because both as_s3_options and opendal lowercase the key again. Second, this method still reads impl, parent and single_level_ns through options.get, which is case-insensitive, so you can end up with the catalog accepting a config that the backend then reports as missing. My preference is to keep the behavior this PR has and add two small things: a warn when a key differs from its canonical form only by case, scoped to the keys the connector owns (impl, parent, parent_delimiter, single_level_ns) plus the first-party prefixes, and a note in docs/src/config.md that keys are case-sensitive. I was going to suggest keeping the original keys and adding a lowercase copy, but I tested it and it does not work. opendal's Configurator::from_iter lowercases keys before handing them to the serde-derived config, and a duplicate field comes back as ConfigInvalid. On 0.57.0, the version this PR pins, passing both Region and region gives duplicate field `region` . OSS and TOS pass the user's keys straight into from_iter (options.clone() at oss.rs:68 and tos.rs:74), and S3/GCS/Azure do the same under use_opendal=true (aws.rs:122), so a doubled map turns keys that work today into hard failures, for example storage.Endpoint on S3 with use_opendal=true. COS, GooseFS and HuggingFace always go through opendal but build their own canonical config map and read exact keys, so they are unaffected. That route needs the duplicate-field problem solved first, or narrowing down to normalizing only the suffixes under the first-party prefixes (storage., credential_vendor., headers., header., tls.).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, keys in docs/src/config.md now have to be copied with their exact case, but the docs don't say so. storage.* at :70, root at :126 and headers.* at :241 each show one spelling only. When it goes wrong the error is Missing required property 'root', and someone looking at their own ROOT won't connect that to casing. A sentence in the docs plus the warn scoped to connector-owned keys should cover it. Worth a look at pushDownFilters and topN_push_down in select.md:245-246 while you're there: they're the two documented keys whose canonical form isn't all lowercase, they go through exact containsKey, and getting the case wrong just silently disables them. Both were already case-sensitive before this change, so that part is a pre-existing trap rather than something this PR introduces.


// Save namespace impl and properties for serialization to workers
this.namespaceImpl = impl;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.lance.spark;

import org.lance.memwal.ShardingSpec;
import org.lance.spark.write.StagedCommit;

import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;

import java.nio.file.Path;
import java.util.HashMap;
import java.util.Map;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

public class BaseLanceNamespaceSparkCatalogInitializeTest {

@TempDir private Path tempDir;

@Test
public void testInitializePreservesCaseSensitiveOptionKeys() {
Map<String, String> rawOptions = new HashMap<>();
rawOptions.put("impl", "dir");
rawOptions.put("root", tempDir.toString());
rawOptions.put("MyMixedCaseKey", "value1");
rawOptions.put("ALLCAPS_KEY", "value2");
rawOptions.put("camelCaseOption", "value3");

CaseInsensitiveStringMap options = new CaseInsensitiveStringMap(rawOptions);

TestCatalog catalog = new TestCatalog();
catalog.initialize("test", options);

Map<String, String> properties = catalog.getNamespaceProperties();

assertTrue(
properties.containsKey("MyMixedCaseKey"),
"Should preserve mixed-case key 'MyMixedCaseKey', got keys: " + properties.keySet());
assertTrue(
properties.containsKey("ALLCAPS_KEY"),
"Should preserve all-caps key 'ALLCAPS_KEY', got keys: " + properties.keySet());
assertTrue(
properties.containsKey("camelCaseOption"),
"Should preserve camelCase key 'camelCaseOption', got keys: " + properties.keySet());

assertEquals("value1", properties.get("MyMixedCaseKey"));
assertEquals("value2", properties.get("ALLCAPS_KEY"));
assertEquals("value3", properties.get("camelCaseOption"));
}

private static class TestCatalog extends BaseLanceNamespaceSparkCatalog {
@Override
public LanceDataset createDataset(
LanceSparkReadOptions readOptions,
StructType sparkSchema,
Map<String, String> initialStorageOptions,
String namespaceImpl,
Map<String, String> namespaceProperties,
boolean managedVersioning,
String fileFormatVersion,
Map<String, String> tableProperties,
ShardingSpec shardingSpec) {
return null;
}

@Override
public LanceDataset createStagedDataset(
LanceSparkReadOptions readOptions,
StructType sparkSchema,
Map<String, String> initialStorageOptions,
String namespaceImpl,
Map<String, String> namespaceProperties,
boolean managedVersioning,
StagedCommit stagedCommit,
String fileFormatVersion,
Map<String, String> tableProperties,
ShardingSpec shardingSpec) {
return null;
}
}
}
Loading