Skip to content

feat: add https connection option to allow plain HTTP transport - #588

Merged
vuanhphung merged 8 commits into
mainfrom
feat/issue-503-http-transport
Nov 13, 2026
Merged

vuanhphung merged 8 commits into
mainfrom
feat/issue-503-http-transport

Conversation

@vuanhphung

Copy link
Copy Markdown
Collaborator

DBSQLClient.getConnectionOptions() hardcoded https: true, so apps that reach Databricks through a trusted local reverse proxy (app --HTTP--> proxy --HTTPS--> Databricks) had to monkey-patch that private method, even though HttpConnection already supports plain HTTP.

This adds a public opt-in, https?: boolean, to ConnectionOptions. It matches the internal IConnectionOptions.https it maps to. HTTPS stays the default, and only an explicit false turns it off. With https: false the default port is 80, and clientCert/clientKey are rejected instead of being silently dropped (the kernel rejects that combination too). The kernel binding keeps an explicit http:// host, so the kernel backend gets http://host[:port]. That also keeps Thrift's Reyden fallback to the kernel on the same transport. On Thrift the scope is deliberately limited to the SQL connection: OAuth, token federation, telemetry and feature-flag requests still use HTTPS. CloudFetch presigned URLs are untouched. CONNECTION_PARAMETERS.md documents this and warns that plain HTTP is only for trusted local or private links.

Verification: new unit tests cover the option mapping and run a real open/close round trip against a local plain-HTTP Thrift server. Without the fix, that round trip fails with a TLS wrong version number error. A new e2e spec runs SELECT 1 on a live warehouse over https: false through a local reverse proxy on both backends. The existing proxy e2e test now uses the public option instead of overriding the private method. On the default path, a live 200k-row query on both backends confirmed Thrift still uses an https.Agent on port 443 and the kernel host is passed through unchanged. npm test (1537 passing), lint, type-check and build pass. Performance isn't affected: the change runs once per connect() while building options, and no per-request or per-row code changed.

Fixes #503

This pull request and its description were written by Isaac.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low-severity edge case. On the kernel path with https: false, setting a port both in host and in port produces a doubled port in the URL. The mapping is correct otherwise: Thrift defaults, the kernel http:// host and rejecting mTLS over plain HTTP all work, and mTLS is rejected on both backends because the kernel path also calls createConnectionProvider.

Comment thread lib/kernel/KernelAuth.ts Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity edge case in how the kernel builds the host URL for bare IPv6 hosts. The option mapping, the mTLS rejection, the docs and the tests (including a real plain-HTTP Thrift round trip) match the intended behaviour. The Thrift-only mTLS check is fine because the kernel enforces it itself.

Comment thread lib/kernel/KernelAuth.ts Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good: one low-severity concern about a trailing slash in the kernel http:// host. The https option maps cleanly onto HttpConnection's existing plain-HTTP support. The defaults stay HTTPS-only, mTLS is rejected up front, and both backends have unit and e2e coverage.

Comment thread lib/kernel/KernelAuth.ts Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

@bot https: false also changes the agent reused by CloudFetch and OAuth to http.Agent. Their HTTPS requests then fail with ERR_INVALID_PROTOCOL, even though their URLs remain unchanged. Could we select the agent by each request’s protocol and cover HTTP SQL transport alongside HTTPS CloudFetch/OAuth?

@peco-engineer-bot

Copy link
Copy Markdown
Contributor

Pushed 2fe61fd to feat/issue-503-http-transport addressing your request.

Started from 6e5c655. Fixed: getAgent(url?) now picks the agent from the URL's protocol, while https only sets the SQL agent. CloudFetch, staging, telemetry and feature flags pass their URLs; OAuth chooses per request. Proxy behavior is unchanged. New tests cover HTTP SQL with HTTPS CloudFetch and OAuth; without the fix they hit ERR_INVALID_PROTOCOL. The PR's mock server answered 500 here (generated code needs Q), so its handlers are now callback-style. npm test (1545), type-check, prettier pass. Lint can't run: airbnb-base is missing. Human needed: delete the empty tests/unit/tmp_debug.test.ts (I can't remove files). E2E not run.

peco-engineer-bot Bot added a commit that referenced this pull request Nov 13, 2026
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
vuanhphung pushed a commit that referenced this pull request Nov 13, 2026
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
@vuanhphung
vuanhphung force-pushed the feat/issue-503-http-transport branch from 2fe61fd to 4657692 Compare November 13, 2026 06:16

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity cleanup: a stray empty debug test file. getAgent now picks the agent from each request's protocol, and every caller in lib/ passes its URL except the Thrift connection, which correctly uses the SQL-transport default. That keeps CloudFetch, OAuth, telemetry and feature-flag requests on HTTPS when https: false is set, and the new unit tests cover this.

Other findings

  • 🔵 Low — This PR adds an empty file, tests/unit/tmp_debug.test.ts (blob e69de29b is the empty blob). It looks like a leftover from local debugging. It has no tests, but it matches the unit test glob, so mocha loads it on every run and it stays in the tree as noise. Please delete it before merging.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

@bot Kernel host already supports full URLs. With { host: 'http://localhost:3015', https: false }, this returns http://http://localhost:3015; an https:// host gets double-prefixed too. Adding the new option therefore breaks these configurations. Please normalize the existing scheme before applying HTTP and checking for an embedded port, and add tests for both URL forms.

@peco-engineer-bot

Copy link
Copy Markdown
Contributor

Pushed db27afe to feat/issue-503-http-transport addressing your request.

Started from 4657692. Fixed in buildKernelHostName (lib/kernel/KernelAuth.ts): with https: false it now strips any existing http:// or https:// scheme (case-insensitive) first, then the trailing /, then checks for an embedded port and applies http://. So 'http://localhost:3015' and 'https://localhost:3015' both become 'http://localhost:3015', with no doubled port. Hosts are unchanged when https isn't false. Added tests in tests/unit/kernel/connectionOptions.test.ts for both URL forms, with and without port, upper case, trailing slash and IPv6. Build, type-check, prettier and npm test pass (1684). npm run lint failed: ESLint can't find the airbnb-base config in this runner. Kernel e2e not run (no warehouse or native build).

peco-engineer-bot Bot added a commit that referenced this pull request Nov 13, 2026
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
DBSQLClient hardcoded `https: true`, so callers behind a trusted local
reverse proxy had to monkey-patch the private getConnectionOptions().
`https: false` now selects plain HTTP (port 80 by default) on the Thrift
transport, and the kernel backend receives an explicit http://host[:port]
host. HTTPS stays the default; mTLS client identity is rejected over HTTP.

Fixes #503

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
vuanhphung and others added 7 commits November 13, 2026 06:33
With `https: false`, the kernel host name appended `port` even when `host`
already carried a `:port` suffix (the kernel's documented convention),
producing an invalid `http://localhost:3015:3015`.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
`/:\d+$/` matched literals like `::1`, so the kernel dropped `port`. Only
`name:port` and `[ipv6]:port` now count as an embedded port, giving the
same URL shape as the Thrift transport.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Thrift strips a trailing `/` from `host` before adding the port; the
kernel mapping did not, producing `http://localhost/:3015`.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity issue: a stray empty debug test file was committed. The agent is now picked by each request's protocol, so CloudFetch, OAuth, telemetry, feature-flag and staging requests keep HTTPS while the SQL transport uses plain HTTP. That change is consistent across all getAgent callers and has unit-test coverage.

Other findings

  • 🔵 Low — This PR adds tests/unit/tmp_debug.test.ts as an empty file. Going by its name, it's a leftover debug scratch file. It has no tests, but mocha's unit glob still picks it up, and it leaves a stray tmp_ file in the repo. Please remove it from the PR.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 13, 2026
@vuanhphung vuanhphung added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 13, 2026
Merged via the queue into main with commit f55509c Nov 13, 2026
33 checks passed
@vuanhphung
vuanhphung deleted the feat/issue-503-http-transport branch November 13, 2026 07:31

This branch was previously deployed

1 inactive deployment
azure-prod — 192a662e Deployed Nov 13, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #1980
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow selecting HTTP transport in DBSQLClient.connect()

1 participant