feat: add https connection option to allow plain HTTP transport

CHANGELOG.md

@@ -7,6 +7,7 @@
 - Thrift HTTP errors (`THTTPException`) now append server detail from the error headers (at most 500 characters). Kernel-backend errors now carry a non-enumerable `cause` summarizing the kernel error code and message (also at most 500 characters). Reyden fallback preserves KP001 after any existing kernel cause. (#467)
 - Add `Symbol.asyncDispose` to clients, sessions, and operations, delegating to `close()`. Use `asyncDisposable()` with interface-typed results in TypeScript; existing interfaces and factory return types remain unchanged. (#331)
 - Thrift `ExecuteStatement` requests are now retried when the connection fails before the request is sent (`ECONNREFUSED`, `ENOTFOUND`, `EAI_AGAIN`), within the existing `retryMaxAttempts` / `retriesTimeout` budget; once retries run out, the original connection error is thrown. Failures after the request may have reached the server are still not retried, so a statement can't run twice. (#458)
+- Add the `https` connection option: set `https: false` to connect to `host` over plain HTTP (port 80 by default), for example through a trusted local reverse proxy. HTTPS remains the default, and plain HTTP should only be used on a trusted local or private link. (#503)
 - CloudFetch retry-exhaustion errors now identify the download and the last failure: the `RetryError` message reads `CloudFetch result download failed: <original message> (last error: <cause>)`. `<cause>` is `HTTP <status>: <statusText>` (or `HTTP <status>` when the status text is empty) for HTTP failures, and `network error <code>`, `network error (<type>)` (e.g. `aborted`, when there is no `code`) or a bare `network error` for network failures (the raw network-error message, which can contain the presigned URL, is not copied). If no cause can be determined, the ` (last error: …)` suffix is omitted. The `RetryError` instance, `errorCode` and `payload` (including the node-fetch `Request` / `Response`) are unchanged. (#471)
 - Report untyped NULL columns as STRING in kernel result metadata, matching Thrift. (#470)
 - `IOperation.close()` no longer throws when the close request fails after the operation has finished (for example, once its results were read, or when the execute response already reported completion): the failure is logged as a warning and reported in the returned status's `info`, and the operation is still closed locally. Closing an operation that hasn't finished still throws on failure. (#480)

CONNECTION_PARAMETERS.md

@@ -42,7 +42,8 @@ backend (the default), the **SEA / Kernel** backend, or both.
 | ----------------------- | --------- | :----: | :----: | ------------- | -------------------------------------------------------------------------------------- |
 | `host`                  | `string`  |   ✅   |   ✅   | — (required)  | Required on both.                                                                      |
 | `path`                  | `string`  |   ✅   |   ✅   | — (required)  | Kernel also derives `x-databricks-org-id` from a `?o=<id>` query parameter.            |
-| `port`                  | `number`  |   ✅   |   ❌   | `443` / —     | Kernel ignores this field. Include a non-default port in `host` for Kernel.            |
+| `port`                  | `number`  |   ✅   |   ⚠️   | `443` / —     | Kernel uses it only with `https: false`; otherwise put a non-default port in `host`.   |
+| `https`                 | `boolean` |   ✅   |   ✅   | `true`        | `false` selects plain HTTP; see [Plain HTTP transport](#plain-http-transport).         |
 | `userAgentEntry`        | `string`  |   ✅   |   ✅   | —             | Folded into the connector `User-Agent` on both.                                        |
 | `useKernel` (internal)  | `boolean` |   —    |   —    | `false`       | Selects the backend: `false`/omitted → Thrift, `true` → Kernel.                        |
 | `clientId` (deprecated) | `string`  |   ✅   |   ✅   | —             | Runtime-only alias for `userAgentEntry`. `userAgentEntry` wins when both are supplied. |
@@ -101,6 +102,34 @@ disabling verification.
 | `clientKey`                                 | `Buffer \| string` |   ✅   |   ✅   | —             | Public mTLS private key. Must be paired with `clientCert`; PKCS#8 is recommended for Kernel portability.                                      |
 | `clientCertPem` / `clientKeyPem` (internal) | `Buffer \| string` |   ❌   |   ✅   | —             | Runtime-only Kernel aliases for the public mTLS pair. They take precedence over `clientCert`/`clientKey` when both pairs are supplied.        |
 
+## Plain HTTP transport
+
+Set `https: false` to connect to `host` over plain HTTP, for example through a
+trusted local reverse proxy that forwards to Databricks over HTTPS (and may
+inject credentials, with `authType: 'custom'` returning no headers):
+
+```ts
+await client.connect({
+  host: 'localhost',
+  port: 3015,
+  path: '/sql/1.0/warehouses/<warehouse-id>',
+  https: false,
+  authType: 'custom',
+  provider: { authenticate: async () => ({}) },
+});
+```
+
+Plain HTTP is unencrypted: SQL text, results, and any credentials the driver
+sends can be read or altered on the network. Use it only on a trusted local or
+private link. With `https: false`, `port` defaults to `80`, the TLS options
+above have no effect, and `clientCert`/`clientKey` are rejected.
+
+On Thrift, `https: false` applies to the SQL connection only. OAuth
+(`databricks-oauth`), token federation, telemetry, and feature-flag requests
+still use HTTPS to `host`. Kernel uses `http://host[:port]` for all of its
+requests, including its own OAuth and telemetry calls. CloudFetch downloads use
+the presigned URLs returned by the server on both backends.
+
 ## Results and type rendering
 
 | Option                        | Type      | Thrift | Kernel | Default Value | Note                                                                                                                                                                               |

lib/DBSQLClient.ts

@@ -231,6 +231,15 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I
       );
     }
 
+    // Only an explicit `false` opts out of TLS.
+    const https = options.https !== false;
+    // http.Agent would silently drop the client identity; the kernel rejects it too.
+    if (!https && hasClientCert) {
+      throw new HiveDriverError(
+        'DBSQLClient: mutual TLS (`clientCert`/`clientKey`) requires HTTPS; remove them or do not set `https: false`.',
+      );
+    }
+
     // Validate the PEM inputs up front with the same ordered BEGIN…END check the
     // kernel path uses (normalizePemBytes), so a truncated/headerless/DER blob is
     // rejected here with a named, actionable error instead of surfacing as an
@@ -246,9 +255,9 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I
 
     return {
       host: options.host,
-      port: options.port || 443,
+      port: options.port || (https ? 443 : 80),
       path: prependSlash(options.path),
-      https: true,
+      https,
       socketTimeout: options.socketTimeout,
       proxy: options.proxy,
       // `customCaCert` is ADDITIVE: Node's `ca` option replaces the system trust

lib/DBSQLSession.ts

@@ -171,7 +171,7 @@ export default class DBSQLSession implements IDBSQLSession {
     }
 
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    const agent = await connectionProvider.getAgent(presignedUrl);
 
     const response = await fetch(presignedUrl, { method: 'GET', headers, agent });
     if (!response.ok) {
@@ -184,7 +184,7 @@ export default class DBSQLSession implements IDBSQLSession {
 
   private async handleStagingRemove(presignedUrl: string, headers: HeadersInit): Promise<void> {
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    const agent = await connectionProvider.getAgent(presignedUrl);
 
     const response = await fetch(presignedUrl, { method: 'DELETE', headers, agent });
     // Looks that AWS and Azure have a different behavior of HTTP `DELETE` for non-existing files.
@@ -209,7 +209,7 @@ export default class DBSQLSession implements IDBSQLSession {
     }
 
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    const agent = await connectionProvider.getAgent(presignedUrl);
 
     const fileStream = fs.createReadStream(localFile);
     const fileInfo = fs.statSync(localFile, { bigint: true });

lib/FeatureFlagCache.ts

@@ -234,7 +234,7 @@ export default class FeatureFlagCache {
   /** Retries transient failures using the connection's retry policy. */
   private async fetchWithRetry(url: string, init: RequestInit): Promise<Response> {
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    const agent = await connectionProvider.getAgent(url);
     const retryPolicy = await connectionProvider.getRetryPolicy();
     const requestConfig: RequestInit = { agent, ...init };
     const result = await retryPolicy.invokeWithRetry(() => {

lib/connection/auth/DatabricksOAuth/OAuthManager.ts

@@ -36,6 +36,8 @@ export default abstract class OAuthManager {
 
   protected agent?: http.Agent;
 
+  protected httpAgent?: http.Agent;
+
   protected issuer?: Issuer;
 
   protected client?: BaseClient;
@@ -60,10 +62,13 @@ export default abstract class OAuthManager {
     // Obtain http agent each time when we need an OAuth client
     // to ensure that we always use a valid agent instance
     const connectionProvider = await this.context.getConnectionProvider();
-    this.agent = await connectionProvider.getAgent();
+    // openid-client picks `http`/`https` by each request's URL, so the agent must
+    // match that URL's protocol rather than the SQL transport's.
+    this.agent = await connectionProvider.getAgent('https:');
+    this.httpAgent = await connectionProvider.getAgent('http:');
 
-    const getHttpOptions = () => ({
-      agent: this.agent,
+    const getHttpOptions = (url?: { protocol: string }) => ({
+      agent: url?.protocol === 'http:' ? this.httpAgent : this.agent,
     });
 
     if (!this.issuer) {

lib/connection/connections/HttpConnection.ts

@@ -32,6 +32,14 @@ const DRIVER_MANAGED_HEADERS = new Set([
 const VALID_HEADER_NAME = /^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$/;
 const INVALID_HEADER_VALUE_CHAR = /[^\t\x20-\x7e\x80-\xff]/;
 
+function getProtocol(url: string | URL): string {
+  if (typeof url !== 'string') {
+    return url.protocol;
+  }
+  const match = /^([a-z][a-z0-9+.-]*:)/i.exec(url);
+  return match ? match[1].toLowerCase() : '';
+}
+
 export default class HttpConnection implements IConnectionProvider {
   private readonly options: IConnectionOptions;
 
@@ -44,7 +52,11 @@ export default class HttpConnection implements IConnectionProvider {
 
   private connection?: ThriftHttpConnection;
 
-  private agent?: http.Agent;
+  private proxyAgent?: http.Agent;
+
+  private httpAgent?: http.Agent;
+
+  private httpsAgent?: https.Agent;
 
   constructor(options: IConnectionOptions, context: IClientContext) {
     this.options = options;
@@ -104,16 +116,33 @@ export default class HttpConnection implements IConnectionProvider {
     return result;
   }
 
-  public async getAgent(): Promise<http.Agent | undefined> {
-    if (!this.agent) {
-      if (this.options.proxy !== undefined) {
-        this.agent = this.createProxyAgent(this.options.proxy);
-      } else {
-        this.agent = this.options.https ? this.createHttpsAgent() : this.createHttpAgent();
+  /**
+   * Returns the agent for a request to `url` (an absolute URL or a bare
+   * protocol such as `'https:'`). Node rejects an agent whose protocol differs
+   * from the request's, so the agent follows the URL rather than `https`, which
+   * only selects the SQL transport. Without `url`, returns the SQL transport's agent.
+   */
+  public async getAgent(url?: string | URL): Promise<http.Agent | undefined> {
+    if (this.options.proxy !== undefined) {
+      // ProxyAgent handles both protocols itself.
+      if (!this.proxyAgent) {
+        this.proxyAgent = this.createProxyAgent(this.options.proxy);
       }
+      return this.proxyAgent;
     }
 
-    return this.agent;
+    const useHttps = url === undefined ? Boolean(this.options.https) : getProtocol(url) !== 'http:';
+    if (useHttps) {
+      if (!this.httpsAgent) {
+        this.httpsAgent = this.createHttpsAgent();
+      }
+      return this.httpsAgent;
+    }
+
+    if (!this.httpAgent) {
+      this.httpAgent = this.createHttpAgent();
+    }
+    return this.httpAgent;
   }
 
   private getAgentDefaultOptions(): http.AgentOptions {

lib/connection/contracts/IConnectionProvider.ts

@@ -10,7 +10,11 @@ export interface HttpTransactionDetails {
 export default interface IConnectionProvider {
   getThriftConnection(): Promise<any>;
 
-  getAgent(): Promise<http.Agent | undefined>;
+  /**
+   * Returns the agent for a request to `url`, whose protocol it must match.
+   * Without `url`, returns the agent for the SQL transport.
+   */
+  getAgent(url?: string | URL): Promise<http.Agent | undefined>;
 
   setHeaders(headers: HeadersInit): void;
 

lib/contracts/IDBSQLClient.ts

@@ -81,6 +81,20 @@ export type ConnectionOptions = {
    */
   escapeMetadataCatalogName?: boolean;
 
+  /**
+   * Connect to `host` over HTTPS (`true`, the default) or plain HTTP (`false`).
+   *
+   * Plain HTTP is unencrypted: SQL, results, and any credentials the driver
+   * sends can be read and modified on the network. Use `false` only for a
+   * trusted local or private link, such as a reverse proxy that forwards to
+   * Databricks over HTTPS. With `false`, `port` defaults to 80, the TLS options
+   * have no effect, and `clientCert`/`clientKey` are rejected.
+   *
+   * On the Thrift backend this applies to the SQL connection only; OAuth, token
+   * federation, telemetry, and feature-flag requests still use HTTPS.
+   */
+  https?: boolean;
+
   /**
    * Verify the server's TLS certificate on the primary Thrift transport.
    * Secure-by-default: omitting this leaves full chain + hostname verification

lib/kernel/KernelAuth.ts

@@ -830,6 +830,22 @@ export function buildKernelProxyOptions(options: ConnectionOptions): KernelProxy
   return { proxy: out };
 }
 
+// The kernel prepends `https://` to a bare host but keeps an explicit scheme.
+// It otherwise ignores `port`, so carry it into the URL unless `host` already has one.
+function buildKernelHostName(options: ConnectionOptions): string {
+  if (options.https !== false) {
+    return options.host;
+  }
+  // The kernel accepts a full URL in `host`, so drop an existing `http://` or
+  // `https://` scheme before applying HTTP (`https: false` selects the scheme).
+  // Then strip a trailing `/` like the Thrift URL builder does.
+  const host = options.host.replace(/^https?:\/\//i, '').replace(/\/$/, '');
+  // `name:port` or `[ipv6]:port`; a bare IPv6 literal like `::1` has no port.
+  const hasPort = /^(\[[^\]]+\]|[^:]+):\d+$/.test(host);
+  const port = options.port && !hasPort ? `:${options.port}` : '';
+  return `http://${host}${port}`;
+}
+
 export function buildKernelConnectionOptions(options: ConnectionOptions): KernelNativeConnectionOptions {
   const { authType } = options as { authType?: string };
 
@@ -843,7 +859,7 @@ export function buildKernelConnectionOptions(options: ConnectionOptions): Kernel
     KernelHttpOptions &
     KernelProxyOptions &
     KernelFederationOptions = {
-    hostName: options.host,
+    hostName: buildKernelHostName(options),
     httpPath: prependSlash(options.path),
     // Match the NodeJS Thrift driver, which surfaces INTERVAL columns as
     // strings. The kernel defaults to native Arrow interval/duration types;

lib/result/CloudFetchResultHandler.ts

@@ -148,7 +148,11 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa
 
   private async fetch(url: RequestInfo, init?: RequestInit): Promise<{ response: Response; body?: ArrayBuffer }> {
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    let requestUrl = url;
+    if (typeof requestUrl !== 'string') {
+      requestUrl = 'href' in requestUrl ? requestUrl.href : requestUrl.url;
+    }
+    const agent = await connectionProvider.getAgent(requestUrl);
     const retryPolicy = await connectionProvider.getRetryPolicy();
 
     const requestConfig: RequestInit = { agent, ...init };

lib/telemetry/DatabricksTelemetryExporter.ts

@@ -332,7 +332,7 @@ export default class DatabricksTelemetryExporter {
 
   private async sendRequest(url: string, init: RequestInit): Promise<Response> {
     const connectionProvider = await this.context.getConnectionProvider();
-    const agent = await connectionProvider.getAgent();
+    const agent = await connectionProvider.getAgent(url);
     // Do NOT route through `connectionProvider.getRetryPolicy()` here. That
     // policy is bounded by the customer's `retriesTimeout` (15 min default)
     // and `retryMaxAttempts` (5), so each call to `sendRequest` could spend

tests/e2e/http_transport.test.ts

@@ -0,0 +1,81 @@
+import { expect } from 'chai';
+import http from 'http';
+import { AddressInfo, Socket } from 'net';
+import httpProxy from 'http-proxy';
+import { DBSQLClient } from '../../lib';
+import { ConnectionOptions } from '../../lib/contracts/IDBSQLClient';
+import { InternalConnectionOptions } from '../../lib/contracts/InternalConnectionOptions';
+
+import config from './utils/config';
+
+// Application --HTTP--> local reverse proxy --HTTPS--> Databricks
+describe('Plain HTTP transport through a local reverse proxy', () => {
+  const requests: Array<{ method?: string; url?: string }> = [];
+  const sockets = new Set<Socket>();
+  let server: http.Server;
+  let port: number;
+
+  before(async () => {
+    const proxy = httpProxy.createProxyServer({ target: `https://${config.host}`, changeOrigin: true, secure: true });
+    server = http.createServer((req, res) => {
+      requests.push({ method: req.method, url: req.url });
+      proxy.web(req, res, {}, (error) => {
+        res.writeHead(502);
+        res.end(String(error));
+      });
+    });
+    server.on('connection', (socket: Socket) => {
+      sockets.add(socket);
+      socket.on('close', () => sockets.delete(socket));
+    });
+    await new Promise<void>((resolve) => {
+      server.listen(0, '127.0.0.1', resolve);
+    });
+    port = (server.address() as AddressInfo).port;
+  });
+
+  after(async () => {
+    const closed = new Promise<void>((resolve) => {
+      server.close(() => resolve());
+    });
+    sockets.forEach((socket) => socket.destroy());
+    await closed;
+  });
+
+  beforeEach(() => {
+    requests.length = 0;
+  });
+
+  [false, true].forEach((useKernel) => {
+    it(`runs a query with https: false (${useKernel ? 'kernel' : 'thrift'} backend)`, async () => {
+      const client = new DBSQLClient();
+      await client.connect({
+        host: '127.0.0.1',
+        port,
+        path: config.path,
+        token: config.token,
+        https: false,
+        useKernel,
+      } as ConnectionOptions & InternalConnectionOptions);
+
+      const session = await client.openSession({ initialCatalog: config.catalog, initialSchema: config.schema });
+      const operation = await session.executeStatement('SELECT 1 AS one');
+      const rows = (await operation.fetchAll()) as Array<Record<string, unknown>>;
+      await operation.close();
+      await session.close();
+      await client.close();
+
+      expect(rows).to.have.length(1);
+      expect(Number(rows[0].one)).to.equal(1);
+      // The SQL traffic went through the plain-HTTP proxy: Thrift to the
+      // warehouse path, the kernel to the SQL Statement Execution API.
+      if (useKernel) {
+        expect(requests.some((request) => request.url?.startsWith('/api/2.0/sql/'))).to.be.true;
+      } else {
+        expect(requests.length).to.be.gte(1);
+        const path = config.path.startsWith('/') ? config.path : `/${config.path}`;
+        expect(requests.every((request) => request.url === path)).to.be.true;
+      }
+    });
+  });
+});