pd.read_parquet("gs://...") fails with CURL error 56 when HTTPS_PROXY is set, pyarrow 23 C++ GCS client ignores NO_PROXY

[dipk-mish7]

We recently upgraded from pyarrow=12 to pyarrow=23 and started seeing this error when reading parquet files from GCS in environments where HTTPS_PROXY is set:

OSError: google::cloud::Status(UNAVAILABLE: Retry policy exhausted ...
PerformWork() - CURL error [56]=Failure when receiving data from the peer)

After investigating, we found that setting empty storage option was letting me read the file

df = pd.read_parquet("gs://bucket/file.parquet", storage_options={})

Questions:

1. Is this a known limitation of the C++ GCS client that it ignores NO_PROXY?
2. Is there any way to make pd.read_parquet("gs://...") honour NO_PROXY without requiring a code change (e.g. an env var or config option)?
3. Should storage_options={} be considered the recommended pattern going forward on pyarrow 13+?

[gitsult4n]

Two separate facts stack up here

NO_PROXY really is dropped on the C++ path
Arrow's GcsFileSystem goes through google-cloud-cpp and its curl layer unconditionally does

handle_.SetOption(CURLOPT_NOPROXY, "metadata.google.internal");

Once CURLOPT_NOPROXY is set explicitly libcurl skips its own no_proxy/NO_PROXY env lookup entirely https_proxy detection still runs separately so every GCS request is forced through the proxy hence curl error 56
GcsOptions in gcsfs.h and the pyarrow GcsFileSystem binding both have zero proxy fields so there is no pyarrow side override for this

storage_options={} did not fix pyarrow it swapped the backend
Pandas only takes the native arrow path when the option is exactly None

if storage_options is None:
    fs, path_or_handle = pa_fs.FileSystem.from_uri(path)

{} is not None so pandas falls through to fsspec.core.url_to_fs(...) which is gcsfs a pure python client that honors NO_PROXY normally You were never fixing arrow you were routing around it

So

• yes known and structural not a bug you can flag
• no env var or pyarrow option restores it
• the supported fix is exactly what you found just be explicit about it

import gcsfs
pd.read_parquet("gs://bucket/path", filesystem=gcsfs.GCSFileSystem())

Passing the filesystem directly is more stable than relying on the storage_options={} side effect since a future pandas release could change that branching