Question: Bazel downloader should only be reading packages, versions, and hashes from pip.parse.requirements_lock, correct?

[dougthor42]

We have a requirements lock file requirements.txt. It contains args for --index-url and --extra-index-url like so:

# This file was autogenerated by uv ...
#    uv pip compile ...
--index-url https://pypi.org/simple
--extra-index-url=https://oauth2accesstoken@baz.com/simple

nurpc==1.0.0 \
   --hash=sha256:...

And our MODULE.bazel has:

pip.parse(
    experimental_index_url = "https://pypi.org/simple",
    experimental_index_url_overrides = {"package": "https//baz.com/simple", ...},  # note no "oauth2accesstoken"
    requirements_lock = "requirements.txt",
    ...

At this point Bazel fails for 401 unauthorized (which is why I'm down this rabbit hole in the first place... you can ignore this):

$ bazel clean --expunge_async; bazel run //:gazelle_python_manifest.update
...
===== stdout start =====                                                                                                                                                                                                                      
Looking in indexes: https://pypi.org/simple, https://****@baz.com/simple                                                                                                                                   
===== stdout end =====                                                                                                                                                                                                                        
===== stderr start =====                                                                                                                                                                                                                      
WARNING: 401 Error, Credentials not correct for https://baz.com/simple/nurpc/                                                                                                                              
ERROR: Could not find a version that satisfies the requirement nurpc==1.0.0 (from versions: none)                                                                                                                                             
ERROR: No matching distribution found for nurpc==1.0.0
...

But if I change requirements.txt:

---extra-index-url https://oauth2accesstoken@baz.com/simple
+--extra-index-url https://baz.com/simple

I get a different Bazel error. Whaaaaaat?:

$ bazel clean --expunge_async; bazel run //:gazelle_python_manifest.update
...
===== stdout start =====                                                                                                                                                                                                                      
Looking in indexes: https://pypi.org/simple, https://baz.com/simple                
User for baz.com:                                                                                                                                                                                                             
===== stdout end =====                                                                                                                                                                                                                        
===== stderr start =====                                                                                                                                                                                                                      
ERROR: Exception:                                                                                                                                                                                                                             
Traceback (most recent call last):
...

Thus, it seems like Bazel may be reading --extra-index-url from the requirements.txt file. Is this intended? If so, why?

Given that the different error is asking for username and password, I'm inclined to think it's related to the credential helper. Our credential helper essentially just calls echo "{headers: {Bearer $(gcloud auth application-default print-access-token)} }", so that isn't reading requirements.txt.

[aignas]

The index interaction happens in 2 places:

• The bazel downloader in the pip.parse extension evaluation is using it to query the index and download the packages. The first error is most likely because the value we are using is from the pip.parse attribute - should we use the one from the file?
• If you are building from sdist, in the whl_library. It seems that this is the second error that you are getting.

EDIT: If you are content with the answer feel free to convert the issue to a discussion (the best place for questions in general), otherwise we can groom this into a feature/bug ticket. :)

[dougthor42]

The first error is most likely because the value we are using is from the pip.parse attribute - should we use the one from the file?

Hard to say. My initial reaction was to say "no, we should use the value from pip.parse.experimental_*" but I don't have any strong reasoning for that.

If you are building from sdist, in the whl_library. It seems that this is the second error that you are getting.

The nurpc thing being downloaded from the private index is already a wheel, so it shouldn't be building from sdist.

convert the issue to a discussion

Ah, thanks! I wasn't aware we used Discussions (or maybe I forgot...). Done!