Skip to content

Commit 951a995

Browse files
committed
[GR-46867] Add wheel builder scripts to our repository, to collect wheel recipes for the community
PullRequest: graalpython/3078
2 parents b34382f + f0dea96 commit 951a995

14 files changed

+7197
-0
lines changed

.github/workflows/build-act-wheels.yml

Lines changed: 3101 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/build-linux-aarch64-wheels.yml

Lines changed: 844 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/build-linux-amd64-wheels.yml

Lines changed: 844 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/build-macos-aarch64-wheels.yml

Lines changed: 737 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/build-macos-amd64-wheels.yml

Lines changed: 737 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: build-repository
2+
'on': workflow_dispatch
3+
jobs:
4+
build-repo:
5+
runs-on: ubuntu-latest
6+
steps:
7+
- name: Download artifacts for linux-amd64
8+
uses: dawidd6/action-download-artifact@v2
9+
with:
10+
workflow: build-linux-amd64-wheels
11+
- name: Download artifacts for linux-aarch64
12+
uses: dawidd6/action-download-artifact@v2
13+
with:
14+
workflow: build-linux-aarch64-wheels
15+
- name: Download artifacts for macos-amd64
16+
uses: dawidd6/action-download-artifact@v2
17+
with:
18+
workflow: build-macos-amd64-wheels
19+
- name: Download artifacts for macos-aarch64
20+
uses: dawidd6/action-download-artifact@v2
21+
with:
22+
workflow: build-macos-aarch64-wheels
23+
- name: Download artifacts for windows-amd64
24+
uses: dawidd6/action-download-artifact@v2
25+
with:
26+
workflow: build-windows-amd64-wheels
27+
- name: Set up Python
28+
uses: actions/setup-python@v4
29+
with:
30+
python-version: '3.10'
31+
- name: Create repository
32+
run: python ${GITHUB_WORKSPACE}/generate_repository.py
33+
- name: Store repository
34+
uses: umutozd/upload-artifact@5c459179e7745e2c730c50b10a6459da0b6f25db
35+
with:
36+
name: repository
37+
path: repository.zip
38+
if-no-files-found: error
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: build-windows-amd64-wheels
2+
'on':
3+
workflow_dispatch:
4+
inputs:
5+
name:
6+
type: string
7+
description: Pkg to build (empty for all)
8+
required: false
9+
graalpy:
10+
type: string
11+
description: GraalPy download url prefix (empty for default)
12+
required: false
13+
jobs: {}

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ This changelog summarizes major changes between GraalVM versions of the Python
44
language runtime. The main focus is on user-observable behavior of the engine.
55

66
## Version 24.0.0
7+
* We now provide a collection of recipes in the form of GitHub Actions to build popular native extensions on GraalPy. These provide a reproducible way for the community to build native extensions for GraalPy with the correct dependencies. See scripts/wheelbuilder/README.md for details.
78
* Foreign big integers are now supported and work with all `Numeric` operators.
89
* Interop `null` values are now treated as *identical*, not only *equal* to Python's `None`. This means that something like `java.type("java.lang.Object[]")(1)[0] is None` will now return `True`.
910
* Update to Python 3.10.13. This inlines the security and bugfixes from 3.10.8 to 3.10.13.

scripts/wheelbuilder/README.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Scripts to build wheels for GraalPy.
2+
3+
[GraalPy](https://github.com/oracle/graalpython) is compatible with many Python
4+
libraries, including those that extend the Python runtime with native code.
5+
However, implemented in Java and thus binary incompatible with existing
6+
extensions, users of native Python extension libraries such as NumPy, SciPy, or
7+
PyTorch have to build their own binaries when installing these libraries. For
8+
many libraries, this means installing additional build dependencies and sitting
9+
through long and resource-intensive compilation processes.
10+
11+
This project is meant to be a place for the community to collect build recipes
12+
for as many popular packages as possible that can then be built once with
13+
GitHub Actions for each major release of GraalPy.
14+
15+
## How to contribute
16+
17+
There should be only one relevant file, `generate_workflow.py`. In it we
18+
collect `BuildSpec` objects that define how to build a particular package. Many
19+
packages do not need special definitions, just a name and maybe which platforms
20+
to build it for. System package dependencies can be specified by platform where
21+
needed. Dependencies between specs are not strictly necessary, but can reduce
22+
the overall build times and thus resource usage of GitHub Action runners.
23+
24+
Changes to `generate_workflow.py` are reflected in the build specs by running
25+
the file. It creates GitHub Action workflow files, one for each platform, and a
26+
giant one with all jobs.
27+
28+
## How to run this
29+
30+
Many packages use a lot of resources to build, and even those that do not
31+
quickly add up. We have chosen GitHub Action workflows as the cross-platform
32+
specification for how to build packages, but it is infeasible to build all
33+
packages on free GitHub Action runners. There are two ways around this:
34+
35+
### Self-hosted runners
36+
37+
The workflow files we generate declare they want to run on self-hosted runners.
38+
Anyone forking this repository to build packages can add their own runners to
39+
their fork and build packages this way. On Linux you will need Docker or Podman
40+
to use the GitHub Action runner script.
41+
42+
### Running actions locally with nektos/act
43+
44+
[Act](https://github.com/nektos/act) allows running GitHub actions locally. We
45+
can use that to just build all the packages on a local machine:
46+
47+
```
48+
./act --artifact-server-path /tmp/artifacts \
49+
-W .github/workflows/build-act-wheels.yml \
50+
-P self-hosted=-self-hosted \
51+
-P macOS=-self-hosted -P Linux=-self-hosted \
52+
-P X64=-self-hosted -P ARM64=-self-hosted \
53+
--input name=all
54+
```
55+
56+
You can vary the `--input name=` argument to build only a single package.
57+
58+
On Linux you will need Docker or Podman. If you're using Podman, make sure you
59+
are running the system service (e.g. `podman system service -t 0` to run it in
60+
a shell), have symlinked or aliased `docker` to `podman`, and prepared you env
61+
with `export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock` to allow
62+
`act` to pick up where podman is listening.
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.
2+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
3+
#
4+
# The Universal Permissive License (UPL), Version 1.0
5+
#
6+
# Subject to the condition set forth below, permission is hereby granted to any
7+
# person obtaining a copy of this software, associated documentation and/or
8+
# data (collectively the "Software"), free of charge and under any and all
9+
# copyright rights in the Software, and any and all patent rights owned or
10+
# freely licensable by each licensor hereunder covering either (i) the
11+
# unmodified Software as contributed to or provided by such licensor, or (ii)
12+
# the Larger Works (as defined below), to deal in both
13+
#
14+
# (a) the Software, and
15+
#
16+
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
17+
# one is included with the Software each a "Larger Work" to which the Software
18+
# is contributed by such licensors),
19+
#
20+
# without restriction, including without limitation the rights to copy, create
21+
# derivative works of, display, perform, and distribute the Software and make,
22+
# use, sell, offer for sale, import, export, have made, and have sold the
23+
# Software and the Larger Work(s), and to sublicense the foregoing rights on
24+
# either these or other terms.
25+
#
26+
# This license is subject to the following condition:
27+
#
28+
# The above copyright notice and either this complete permission notice or at a
29+
# minimum a reference to the UPL must be included in all copies or substantial
30+
# portions of the Software.
31+
#
32+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
33+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
34+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
35+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
36+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
37+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
38+
# SOFTWARE.
39+
40+
"""
41+
Set up a repository folder following
42+
https://packaging.python.org/en/latest/guides/hosting-your-own-index/
43+
"""
44+
45+
import glob
46+
import gzip
47+
import os
48+
import re
49+
import shutil
50+
import sys
51+
52+
53+
def normalize(name):
54+
"""
55+
Taken from
56+
https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization
57+
"""
58+
return re.sub(r"[-_.]+", "-", name).lower()
59+
60+
61+
if __name__ == "__main__":
62+
if len(sys.argv) > 1:
63+
workspace = sys.argv[1]
64+
else:
65+
workspace = os.environ["GITHUB_WORKSPACE"]
66+
# find wheels either locally in the workspace or in an act artifact root structure
67+
wheels = glob.glob(os.path.join(workspace, "**/*.whl")) or glob.glob(
68+
os.path.join(workspace, "**/**/*.whl.gz__")
69+
)
70+
repo = os.path.join(workspace, "repository")
71+
os.makedirs(repo, exist_ok=True)
72+
for wheel in wheels:
73+
print("Processing", wheel)
74+
parts = os.path.basename(wheel).split("-")
75+
for idx, part in enumerate(parts):
76+
if part.startswith("graalpy3") or part.startswith("py3"):
77+
version_idx = idx - 1
78+
break
79+
wheel_name = normalize("-".join(parts[:version_idx]))
80+
target_dir = os.path.join(repo, wheel_name)
81+
os.makedirs(target_dir, exist_ok=True)
82+
if wheel.endswith(".gz__"):
83+
with gzip.open(wheel, "rb") as f_in:
84+
with open(
85+
os.path.join(
86+
target_dir, os.path.basename(wheel).replace(".gz__", "")
87+
),
88+
"wb",
89+
) as f_out:
90+
shutil.copyfileobj(f_in, f_out)
91+
else:
92+
shutil.copy(wheel, target_dir)
93+
94+
shutil.make_archive(f"{workspace}/repository", "zip", repo)

0 commit comments

Comments
 (0)