Missing files when running with `venvs_site_packages=yes`

[FrankPortman]

When I try to run bazel build @rules_python//python/config_settings:venvs_site_packages=yes //... I see the following output from one of my internal usages of py_wheel:

INFO: Invocation ID: 45e6114d-6e00-443c-95d4-ae1757315abd
ERROR: /Users/frank/workspace/yobi/data-alt-2/tools/runnable/examples/python/BUILD:10:17: Building wheel @@//tools/runnable/examples/python:example_app_deploy_wheel failed: (Exit 1): sandbox-exec failed: error executing PyWheel command
  (cd /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    TMPDIR=/var/folders/kb/_rxq15gs3gq21765gfcbbdjw0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/sandbox.sb /var/tmp/_bazel_fp/install/f96a4af1851d4fb5dafa59d311169545/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/stats.out' bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker --name example_app --version 0 --python_tag py3 --abi none --platform any --out bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app-0-py3-none-any.whl --name_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.name --input_file_list bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_target_wrapped_inputs.txt --metadata_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.metadata.txt --entry_points_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_entry_points.txt)
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 506, in <module>
    main()
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 500, in main
    _run_py_path(main_filename, args=sys.argv[1:])
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 287, in _run_py_path
    runpy.run_path(main_filename, run_name="__main__")
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 660, in <module>
    main()
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 560, in main
    maker.add_file(package_filename, real_filename)
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 323, in add_file
    self._whlfile.add_file(package_filename, real_filename)
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 173, in add_file
    with open(real_filename, "rb") as fsrc:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/_example_app_bin.venv/lib/python3.12/site-packages/attr'
INFO: Elapsed time: 2.497s, Critical Path: 2.02s
INFO: 79 processes: 53 action cache hit, 79 internal.
ERROR: Build did NOT complete successfully

I am still investigating whether this is user error and whether I may have been doing something "unofficial" without declaring formal runfiles deps, but wanted to post this early since I know there is a lot of ongoing work for this feature.

[rickeylev]

Are you putting a py_binary in a py_wheel? It's dubious if that's supported or not.

This sounds similar to the rules_pkg issue where, because the venv uses relative symlinks, handling them needs to be done specially.

In this case, it looks like the wheel packager is trying to read a symlink to a directory as if it was a file. Zips don't officially support storing symlinks, but it's possible with some special extensions. At the least, it could dereference the symlinks at do a full copy.

[FrankPortman]

Are you putting a py_binary in a py_wheel?

Yes - for legacy reasons. And in practice I only care about the PyInfo-esque information. I had to introduce a different heuristic to use bootstrap_impl=script (related to when I opened: #3324).

def _is_runtime_artifact(file):
    path = file.short_path
    basename = file.basename

    # Venv artifacts added by `rules_python` with bootstrap_impl=script
    if path.endswith(".venv/bin/python3"):
        return True

    # Bootstrap scripts added by `rules_python` with bootstrap_impl=script
    if basename.endswith("_stage2_bootstrap.py"):
        return True

    return False

    ...

    inputs = depset(
        transitive = [dep[DefaultInfo].data_runfiles.files for dep in ctx.attr.deps] +
                     [dep[DefaultInfo].default_runfiles.files for dep in ctx.attr.deps],
    )

    # Gather the pypi deps
    direct_pypi_deps = [dep for dep in ctx.attr.deps if PypiInfo in dep]
    indirect_deps = [dep[PypiInfo].pypi_deps for dep in direct_pypi_deps]
    all_deps = depset(direct = direct_pypi_deps, transitive = indirect_deps)

    infos = [
        (dep[PypiInfo].name, dep[PypiInfo].version)
        for dep in all_deps.to_list()
        if hasattr(dep[PypiInfo], "name")
    ]

    internal_deps = [
        dep
        for dep in inputs.to_list()
        if dep.owner.repo_name == _THIS_REPO and not _is_runtime_artifact(dep)
    ]

    ...

I think if my rule is working then it shouldn't even try to copy external files though (it should separate out external packages into tuples of (package, version). I will need to look into this more.

[shayanhoshyari]

@rickeylev I am facing a different instance of Missing files when running with venvs_site_packages=yes with 1.7.0 + linux.

TLDR is that, when I do

py_binary(
    name = "hello_world_app",
    srcs = [
        "app.py",
        "__main__.py"
    ],
    main = "__main__.py",
    deps = [
        "@pip//cowsay",
        "@pip//torch",
    ],
)

• At least one .so is missing bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/nccl//lib/libnccl.so.2
• While the symlink is missing, the root runfile bazel-bin/app.runfiles/rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2 is included.
• the other .py files from this library are where they should be).
• Other .so files seem to be where they should be in th venv, e.g. bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/cufft/lib/libcufft.so.11

If I do however

py_binary(
    name = "hello_world_app",
    srcs = [
        "app.py",
        "__main__.py"
    ],
    main = "__main__.py",
    deps = [
        "@pip//cowsay",
        "@pip//nvidia_nccl_cu12",
    ],
)

the .so appears. If I do both torch and nccl_cu12, the .so goes missing again.

Here is a full repro: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so

What this causes is that if my docker container does not have nccl then import torch completely fails. If it does, but different version, importing torch gives things like this due to version mismatch

    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
undefined symbol: ncclGroupSimulateEnd

Some debugging with this modified code

Debug prints: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1

printed output

Computing main repo mapping: 
Loading: 
Loading: 1 packages loaded
Analyzing: target //:app (2 packages loaded, 0 targets configured)
Analyzing: target //:app (2 packages loaded, 0 targets configured)

DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:300:22: 
	entry: struct(files = depset([<source file site-packages/nvidia/nccl/lib/libnccl.so.2>]), kind = "LIB", link_to_file = <source file site-packages/nvidia/nccl/lib/libnccl.so.2>, link_to_path = "rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/libnccl.so.2", package = "nvidia_nccl_cu12", venv_path = "nvidia/nccl/lib/libnccl.so.2", version = "2.27.5") 
	runfiles_dir_name: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1 
	site_packages_root: site-packages/ 
	filename: libnccl.so.2
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:300:22: 
	entry: struct(files = depset([<source file site-packages/nvidia/cufft/lib/libcufft.so.11>]), kind = "LIB", link_to_file = <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, link_to_path = "rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufft.so.11", package = "nvidia_cufft_cu12", venv_path = "nvidia/cufft/lib/libcufft.so.11", version = "11.3.3.83") 
	runfiles_dir_name: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e 
	site_packages_root: site-packages/ 
	filename: libcufft.so.11
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: cowsay 1
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: cowsay-6.1.dist-info 1
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: nvidia 7
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
	 .so. WINS 
	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/nvjitlink/__init__.py>, <source file site-packages/nvidia/nvjitlink/include/__init__.py>, <source file site-packages/nvidia/nvjitlink/include/nvJitLink.h>, <source file site-packages/nvidia/nvjitlink/lib/__init__.py>, <source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>] 
	entry.venv_path: nvidia 
	rf_root_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
	 .so. WINS 
	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/cufft/__init__.py>, <source file site-packages/nvidia/cufft/include/__init__.py>, <source file site-packages/nvidia/cufft/include/cudalibxt.h>, <source file site-packages/nvidia/cufft/include/cufft.h>, <source file site-packages/nvidia/cufft/include/cufftXt.h>, <source file site-packages/nvidia/cufft/include/cufftw.h>, <source file site-packages/nvidia/cufft/lib/__init__.py>, <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
	entry.venv_path: nvidia 
	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufft.so.11 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
	 .so. WINS 
	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/cufft/__init__.py>, <source file site-packages/nvidia/cufft/include/__init__.py>, <source file site-packages/nvidia/cufft/include/cudalibxt.h>, <source file site-packages/nvidia/cufft/include/cufft.h>, <source file site-packages/nvidia/cufft/include/cufftXt.h>, <source file site-packages/nvidia/cufft/include/cufftw.h>, <source file site-packages/nvidia/cufft/lib/__init__.py>, <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
	entry.venv_path: nvidia 
	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufftw.so.11 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
	 .so. loses
	 entry.files [<source file site-packages/nvidia/cufft/lib/libcufft.so.11>] 
	entry.venv_path: nvidia/cufft/lib/libcufft.so.11 
	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufft.so.11 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufft.so.11
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
	 .so. loses
	 entry.files [<source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
	entry.venv_path: nvidia/cufft/lib/libcufftw.so.11 
	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufftw.so.11 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufftw.so.11
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
	 .so. loses
	 entry.files [<source file site-packages/nvidia/nccl/lib/libnccl.so.2>] 
	entry.venv_path: nvidia/nccl/lib/libnccl.so.2 
	rf_root_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/libnccl.so.2
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
	 .so. loses
	 entry.files [<source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>] 
	entry.venv_path: nvidia/nvjitlink/lib/libnvJitLink.so.12 
	rf_root_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12 
	entry.link_to_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/libnvJitLink.so.12
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: nvidia_cufft_cu12-11.3.3.83.dist-info 1
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: nvidia_nccl_cu12-2.27.5.dist-info 1
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
	group: nvidia_nvjitlink_cu12-12.8.93.dist-info 1
DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:132:14: 
	keep_kind_link_map: {"cowsay": "rules_python++pip+pip_313_cowsay_py3_none_any_274b1e6f/site-packages/cowsay", "cowsay-6.1.dist-info": "rules_python++pip+pip_313_cowsay_py3_none_any_274b1e6f/site-packages/cowsay-6.1.dist-info", "nvidia/__init__.py": <source file site-packages/nvidia/__init__.py>, "nvidia/nvjitlink/__init__.py": <source file site-packages/nvidia/nvjitlink/__init__.py>, "nvidia/nvjitlink/include/__init__.py": <source file site-packages/nvidia/nvjitlink/include/__init__.py>, "nvidia/nvjitlink/include/nvJitLink.h": <source file site-packages/nvidia/nvjitlink/include/nvJitLink.h>, "nvidia/nvjitlink/lib/__init__.py": <source file site-packages/nvidia/nvjitlink/lib/__init__.py>, "nvidia/nvjitlink/lib/libnvJitLink.so.12": <source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>, "nvidia/cufft/__init__.py": <source file site-packages/nvidia/cufft/__init__.py>, "nvidia/cufft/include/__init__.py": <source file site-packages/nvidia/cufft/include/__init__.py>, "nvidia/cufft/include/cudalibxt.h": <source file site-packages/nvidia/cufft/include/cudalibxt.h>, "nvidia/cufft/include/cufft.h": <source file site-packages/nvidia/cufft/include/cufft.h>, "nvidia/cufft/include/cufftXt.h": <source file site-packages/nvidia/cufft/include/cufftXt.h>, "nvidia/cufft/include/cufftw.h": <source file site-packages/nvidia/cufft/include/cufftw.h>, "nvidia/cufft/lib/__init__.py": <source file site-packages/nvidia/cufft/lib/__init__.py>, "nvidia/cufft/lib/libcufft.so.11": <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, "nvidia/cufft/lib/libcufftw.so.11": <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>, "nvidia/nccl/include/nccl.h": <source file site-packages/nvidia/nccl/include/nccl.h>, "nvidia_cufft_cu12-11.3.3.83.dist-info": "rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia_cufft_cu12-11.3.3.83.dist-info", "nvidia_nccl_cu12-2.27.5.dist-info": "rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia_nccl_cu12-2.27.5.dist-info", "nvidia_nvjitlink_cu12-12.8.93.dist-info": "rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia_nvjitlink_cu12-12.8.93.dist-info"}
INFO: Analyzed target //:app (90 packages loaded, 3391 targets configured).
INFO: Found 1 target...
Target //:app up-to-date:
  bazel-bin/app
INFO: Elapsed time: 0.261s, Critical Path: 0.00s
INFO: 1 process: 31 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total action

1. only single nccl works because here we enter the group 1 case. https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L117-L122
2. Other libs somehow work (e.g. CUFFT) because we add multiple VenvSymlinkEntry for them. At

• https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L260-L269
• https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330
• according to prints, the first one never wins due to that startswith failing here: https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330

      .so. loses
  entry.files [<source file site-packages/nvidia/nccl/lib/libnccl.so.2>] 
  entry.venv_path: nvidia/nccl/lib/libnccl.so.2 
  rf_root_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2 
  entry.link_to_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/libnccl.so.2
  

3. some libs like nccl only go to 2.1, and not 2.2 so their links never gets created.

This is on 1.7.0. I see there are changes in these areas in main, so this analysis might be stale there. That being said, why in main I am losing all .so s, that is another question.

[shayanhoshyari]

I tried the head, 9a20fc243a23399511023d78209590f776b36e38 of rules_python with same code, suddenly all .so symlinks went missing :|

[rickeylev]

Great debugging, thank you! I'll have a look. I was manually testing torch + nvidia deps, so hrmph 😅 . I swear a added a unit test exactly for this combination of things >.<

link_to_path = [...]/site-packages/libnccl.so.2 -- because nvidia/nccl/lib is missing in the middle, it probably means something is wrong in the get_venv_symlinks() function. Somehow around L342, where lib*.so* files are handled. Its the only place where an entry like that gets created (one with link_to_file set)

[shayanhoshyari]

@rickeylev Thanks for quick reply. I edited my post #3470 (comment) with the prints I added, and some more info.

I swear a added a unit test exactly for this combination of things

You mean https://github.com/bazel-contrib/rules_python/blob/main/tests/venv_site_packages_libs/shared_lib_loading_test.py right? Looking here a few questions:

1.7.0

1. On 1.7.0, is it intentional that you are making adder.so (without lib) here and libincrement.so? but the check here explicitly looks for lib prefix? I found this out when seeing this entry making does not apply to adder and instead it is saved by this one.
2. In 1.7.0 I can make the unit test break (emulating by what happens to nvidia/nccl) with this change: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1 (the changes in venv_runfiles.bzl is just prints, the failure changes are simply in the test BUILD).

In general in 1.7.0 the logic around adding entries multiple times for same .so file, the second one coming from https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330 makes things hard to debug as somethings pass luckily due to second entry (but not always). E.g. the unit test.

But in theory 41f91e9 (in main and not in 1.7.0 should address the above. Now let me move to main and see what is going on there. I do see missing stuff there too: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so#main

main

😂 ok this was an easy one. issue was here: https://github.com/bazel-contrib/rules_python/blob/main/python/private/venv_runfiles.bzl#L67. when ctx.package is "", i.e. root BUILD, you get an abs path, and messes things up.

Change

symlink_from = "{}/{}".format(ctx.label.package, bin_venv_path)

to

paths.join(ctx.label.package, bin_venv_path) 

fixes it.

Let me raise a PR

[shayanhoshyari]

@rickeylev fix for my issue in main (i.e. #3470 (comment)): #3474

[rickeylev]

Heh, yeah, the logic and code structure in 1.7 and earlier was convoluted. Nice to have that refactor validated 😄 .

is it intentional that you are making adder.so (without lib) here and libincrement.so?

Yes. It's verifying that foo.so (python C modules, not regular C libs) don't get explicitly linked (and thus can be grouped into fewer link operations). From what I could tell, it was only lib*.so* files being processed by the dynamic linker that would compute the wrong LD search path. Python C modules (loaded via python's import system) didn't seem to exhibit that same behavior. LMK if you see otherwise.

[FrankPortman]

@rickeylev re my question in the initial issue - is this impacted by the recent PR in the thread and/or is the missing file WAI?

[rickeylev]

Oh whoops, no, the thing shayanhashyari fixed is something different. I'll reopen this.