"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)

[dougthor42]

🐞 bug report

Affected Rule

Maybe pip.parse?

Is this a regression?

It appears to be, yes. I bisected from 1.6.1 to 1.7.0 and found that commit 7b88c87 (PR #3243) is the first bad commit.

Description

We're getting a "duplicate library" error.

Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py

🔬 Minimal Reproduction

TL;DR: running multiple python versions with the same minor version (eg 3.13.4 and 3.13.6) appears to break things.

I'll see what I can do about making a minimal reproduction.

Background

We have things set up so that each python target is actually a macro wrapping py_* for the current python version and py_* for the "pynext" python version. Example:

load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test")  # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
    _py_binary(**kwargs)
    _py_binary_next(**kwargs)

And thus our MODULE.bazel file has:

PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6"  # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    ...,
    is_default = True,
    python_version = PYTHON_VERSION,
)
python.toolchain(
    ...,
    is_default = False,
    python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    experimental_index_url = ...,
    experimental_index_url_overrides = ...,
    hub_name = "pypi",
    python_version = PYTHON_VERSION,
    requirements_lock = "//:requirements.txt",
)
pip.parse(
    experimental_index_url = ...,
    experimental_index_url_overrides = ...,
    hub_name = "pypi",
    python_version = PYNEXT_VERSION,
    requirements_lock = "//:requirements.txt",
)

You'll notice that we're using the same requirements.txt lock file and hub_name for both current and "pynext" versions.

We then use build/test tags to only build/test the current python version or the "pynext" python version:

# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...

# optionally test against both versions
bazel test --test_tag_filters= //...

# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...

🔥 Exception or Error

 bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
                mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
                builder.pip_parse(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
                pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
                _create_whl_repos(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
                _add_whl_library(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
                fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
    currently loading: 
    Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s

🌍 Your Environment

Operating System:

gLinux (Debian Testing)

Output of bazel version:

$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620

Rules_python version:

1.7.0

Anything else relevant?

Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping PYNEXT_VERSION to a different minor version than PYTHON_VERSION appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.

This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:

• Do not build/test/run any "pynext" version by default
• Only need to support versions N and N+1, where N might be a major, minor, or patch version
• Must use the same requirements.txt lock file.
• Must use the same pypi hub name
  • Or more accurately: the third-party deps of a target must not change between python versions.

[aignas]

I don't think we should support this, to be honest because upstream python tooling usually does not have this differentiation.

What would be the use-case here? And I can be convinced otherwise. What is the reason to have multiple patch versions?

[rickeylev]

I think this is possible through a round about way? IIRC, under the hood, we allow arbitrary config settings to map to a particular requirements.txt.

What might work is something like:

• Define a pynext platform using pip.default
• Specify that platform in pip.parse

# File: MODULE.bazel
pip.default(
  platform = "pynext"
  config_settings = ["@//:is_pynext"],
)
pip.parse(
  platforms = ["pynext"],
)

# File: BUILD.bazel
config_setting(
  name = "is_pynext",
  flag_values = {
    "@rules_python//python/config_settings:python_version": "3.13.6",
  }
)

Admittedly, rather awkward. I think pip.default also expects e.g. os, arch, etc, but don't entirely recall.

I guess, ideally, pip.parse would treat python_version = ... as a short hand way of specifying config_settings = [":is_python_..."] ?

Error: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py

Alternatively -- is this just a bug with duplicate repo definitions? If it's trying to create a repo pointing to the same URL, it should just use e.g. first defined wins semantics

[dougthor42]

What would be the use-case [for] ... multiple patch versions?

N.B.: I'm most just playing devil's advocate here. I agree that the 3p wheels/sdists themselves don't differentiate - they also can't, as the {python tag} is set by py_version_nodot which is explicitly set to f'{sys.version_info[0]}{sys.version_info[1]}'.¹ Therefore there shouldn't be any need - from a third-party package perspective - to differentiate between, say, 3.13.4 and 3.13.6.

Some users may have stricter requirements when it comes to bumping python versions. While the ABI between python patch versions is guaranteed to be compatible, bugs can creep in. For example, 3.12.1 introduced a unittest bug that was fixed in 3.12.2.

pip.parse would treat python_version = ... as a short hand way of specifying config_settings = [":is_python_..."] ?

I think this makes sense to me.

Alternatively -- is this just a bug with duplicate repo definitions?

Perhaps. After all, things did work with 1.6.1. Whether or not they were supposed to work is a different story. For the record, I'm fine if I happened to be relying on undefined/incorrect behavior.

---

I wonder if there is some de-duplication opportunity here? If users have multiple python versions defined in MODULE.bazel (or config_settings) and multiple pip.parse hubs, but the net result is the same wheel/binary version, could we reduce the number of fetched things?

For example, flask-3.1.2-py3-none-any.whl is compatible with [any python version >= 3.10](https://github.com/pallets/flask/blob/2579ce9f18e67ec3213c6eceb5240310ccd46af8/pyproject.toml#L22C1-L22C27. If someone has a pip hub for python 3.10, 3.11, 3.12, and 3.13 that all use the same version of flask, that file would be downloaded/processed 4 times, no?

Footnotes

1. References added mostly for my own sanity and future-self. ↩

[aignas]

Regarding deduplication, yeah, once #2949 is verified to be working, we can go and finish #2948 so that we need to carry around fewer things. In the past, it was not possible because the python interpreter was needed to decide the deps closure.

I think in general I am not against supporting the ask here and it should be straight forward once deduplication is done - just add extra config settings in the hub. Before it is done, we need to have the full python version in the whl_library_args in the hub_builder and then the error would not be triggered. The whl_repo_name function should be changed.