[GR-21590] Backport a multitude of native extension fixes.

PullRequest: graalpython/2749

Author: timfel

ci.jsonnet

@@ -1 +1 @@
-{ "overlay": "fe47c77ef9ca3412a9faf4889643535e2cc7d4c9" }
+{ "overlay": "e61df4b2dde96b3c9b488ab7cde6b011d719db79" }

docs/user/Interoperability.md

@@ -38,27 +38,26 @@ You can import a global value from the entire polyglot scope:
 ```
 
 This global value should then work as expected:
+    
 * Accessing attributes assumes it reads from the `members` namespace.
-```python
->>> ruby_polyglot.to_s
-<foreign object at ...>
-```
+    ```python
+    >>> ruby_polyglot.to_s
+    <foreign object at ...>
+    ```
 
-* Calling methods on the result tries to do a straight invoke and falls
-back to reading the member and trying to execute it.
-```python
->>> ruby_polyglot.to_s()
-Polyglot
-```
+* Calling methods on the result tries to do a straight invoke and falls back to reading the member and trying to execute it.
+    ```python
+    >>> ruby_polyglot.to_s()
+    Polyglot
+    ```
 
 * Accessing items is supported both with strings and numbers.
-```python
->>> ruby_polyglot.methods()[10] is not None
-True
-```
+    ```python
+    >>> ruby_polyglot.methods()[10] is not None
+    True
+    ```
 
-You can export some object from Python to other supported languages so they can import
-it:
+You can export some object from Python to other supported languages so they can import it:
 ```python
 >>> foo = object()
 >>> polyglot.export_value(value=foo, name="python_foo")
@@ -77,7 +76,7 @@ In this case the function name is used as the globally exported name:
 ```
 
 Here is an example of how to use the JavaScript regular expression engine to
-match Python strings.
+match Python strings:
 ```python
 >>> js_re = polyglot.eval(string="RegExp()", language="js")
 
@@ -101,7 +100,7 @@ To run this example, first install the required R library:
 R -e 'install.packages("https://www.rforge.net/src/contrib/jpeg_0.1-8.tar.gz", repos=NULL)'
 ```
 
-This example also uses [image_magix.py](http://graalvm.org/docs/examples/image_magix.py) and works
+This example also uses [image_magix.py](https://www.graalvm.org/resources/img/python/image_magix.py) and works
 on a JPEG image input (you can try with [this image](https://www.graalvm.org/resources/img/python_demo_picture.jpg)). These files have to be in the same folder that the script below is located in and executed from.
 ```python
 import polyglot

docs/user/PythonStandaloneBinaries.md

@@ -0,0 +1,59 @@
+---
+layout: docs-experimental
+toc_group: python
+link_title: Python Standalone Applications
+permalink: /reference-manual/python/standalone-binaries/
+---
+# Standalone Applications with Python
+
+With GraalVM Python implementation (GraalPy), you can distribute Python applications or libraries as standalone binaries or JAR files without any external dependencies.
+The [Truffle framework](https://github.com/oracle/graal/tree/master/truffle) that GraalPy is built on, and the [Sulong LLVM runtime](https://github.com/oracle/graal/tree/master/sulong) that GraalPy leverages for managed execution of Python's native extensions enables users to completely virtualize all filesystem accesses of Python programs, including those to the standard library and installed packages.
+
+GraalPy comes with a module that can create standalone binaries or Java project skeletons.
+The binaries bundle everything into one native executable.
+The Java skeletons are set up with Maven to build and run self-contained JAR files.
+They can also be used to generate a standalone binary from those JARs later, so Java skeletons offer more flexibility and control over the steps.
+
+### Prerequisite
+
+Set `JAVA_HOME` to use a GraalVM distribution.
+
+## Creating GraalPy Binaries
+
+Suppose there is a simple Python script, `my_script.py`, that does some useful work when run directly.
+To distribute it as a standalone native binary, run the following command:
+
+```
+graalpy -m standalone binary --module my_script.py --output my_binary
+```
+
+It generates a standalone `my_binary` file which includes the Python code, the GraalPy runtime, and the Python standard library in a single, self-contained executable.
+Use `graalpy -m standalone binary --help` for further options.
+
+## Embedding GraalPy in a Java Application
+
+You can distribute the Python script as a JAR file that runs on GraalVM and includes GraalPy.
+To achieve this, run the `java` subcommand of GraalPy's `standalone` module:
+
+```
+graalpy -m standalone java --output-directory MyJavaApplication --module my_script.py
+```
+
+It creates a Java project _MyJavaApplication_. It includes a `pom.xml` that makes it easy to generate a JAR or a GraalVM native executable with Maven.
+You can open this Maven project with any Java IDE and edit the main class that was created to modify the Python embedding.
+To build the application, either `mvn -Pjar package` to create a JAR file, or `mvn -Pnative package` to create a GraalVM native executable.
+
+Take a look at the generated `pom.xml`.
+There are some options to tweak the performance and footprint trade-off.
+Review the [Python Native Images documentation](PythonNativeImages.md) to find out how to remove other unwanted components and further reduce the binary size.
+
+The generated project should be viewed as a starting point.
+It includes the entire Python standard library, so the Python code can invoke all of the standard library code.
+The resources can be manually pruned to reduce the included Python libraries to the necessary amount, reducing both the size of the package and the time to start up.
+This Java example demonstrates some useful default options for the Python context, but other settings may be desirable to further control what the Python code is allowed to do.
+
+## Security Considerations
+
+Creating a native executable or a JAR that includes the Python code could be seen as a mild form of obfuscation, but it does not protect your source code.
+While the Python sources are not stored verbatim into the image (only the GraalPy bytecode is), that bytecode is easy to convert back into Python sources.
+If stronger protection for the included Python source code is required, consider, for example, encryption of the resources before building the native executable, and adding appropriate decryption into the generated virtual file system.

graalpython/com.oracle.graal.python.cext/include/genobject.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2020, 2022, Oracle and/or its affiliates.
+/* Copyright (c) 2020, 2023, Oracle and/or its affiliates.
  * Copyright (C) 1996-2017 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2

graalpython/com.oracle.graal.python.cext/include/pystrcmp.h

@@ -0,0 +1,28 @@
+/* Copyright (c) 2023, Oracle and/or its affiliates.
+ * Copyright (C) 1996-2022 Python Software Foundation
+ *
+ * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
+ */
+#ifndef Py_STRCMP_H
+#define Py_STRCMP_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+PyAPI_FUNC(int) PyOS_mystrnicmp(const char *, const char *, Py_ssize_t);
+PyAPI_FUNC(int) PyOS_mystricmp(const char *, const char *);
+
+#ifdef MS_WINDOWS
+#define PyOS_strnicmp strnicmp
+#define PyOS_stricmp stricmp
+#else
+#define PyOS_strnicmp PyOS_mystrnicmp
+#define PyOS_stricmp PyOS_mystricmp
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* !Py_STRCMP_H */

graalpython/com.oracle.graal.python.cext/posix/posix.c

@@ -541,14 +541,34 @@ int32_t call_kill(int64_t pid, int32_t signal) {
     return kill(pid, signal);
 }
 
+int32_t call_killpg(int64_t pgid, int32_t signal) {
+    return killpg(pgid, signal);
+}
+
 int64_t call_getuid() {
     return getuid();
 }
 
+int64_t call_getgid() {
+    return getgid();
+}
+
 int64_t call_getppid() {
     return getppid();
 }
 
+int64_t call_getpgid(int64_t pid) {
+    return getpgid(pid);
+}
+
+int32_t call_setpgid(int64_t pid, int64_t pgid) {
+	return setpgid(pid, pgid);
+}
+
+int64_t call_getpgrp() {
+    return getpgrp();
+}
+
 int64_t call_getsid(int64_t pid) {
     return getsid(pid);
 }

graalpython/com.oracle.graal.python.cext/setup.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2019, 2022, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2019, 2023, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # The Universal Permissive License (UPL), Version 1.0
@@ -77,6 +77,7 @@ def setup(*args, **kwargs):
     # wrap the distutil setup. since we're running in the same process, running
     # a full clean will fail the next build, since distutils thinks it already
     # created the "build" directory
+    shutil.rmtree("build", ignore_errors=True)
     os.makedirs("build", exist_ok=False)
     return distutils_setup(*args, **kwargs)
 

graalpython/com.oracle.graal.python.cext/src/abstract.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2018, 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2023, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The Universal Permissive License (UPL), Version 1.0
@@ -56,13 +56,6 @@ static PyObject* null_error(void) {
     return NULL;
 }
 
-int PyIter_Check(PyObject *obj) {
-    PyTypeObject *tp = Py_TYPE(obj);
-    return (tp->tp_iternext != NULL &&
-            tp->tp_iternext != &_PyObject_NextNotImplemented &&
-            ((PyObject *)tp->tp_iternext) != Py_NotImplemented);
-}
-
 static PyObject * do_unaryop(PyObject *v, UnaryOp unaryop) {
     return GraalPyTruffleNumber_UnaryOp(v, (int) unaryop);
 }
@@ -480,16 +473,6 @@ PyObject ** _PySequence_Fast_ITEMS(PyObject *o) {
     return PyList_Check(o) ? PyListObject_ob_item(o) : PyTupleObject_ob_item(o);
 }
 
-int _PyIter_Check(PyObject* obj) {
-	iternextfunc func = PyTypeObject_tp_iternext(Py_TYPE(obj));
-	return func != NULL && func != &_PyObject_NextNotImplemented;
-
-}
-int _PyIndex_Check(PyObject* obj) {
-	PyNumberMethods* methods = PyTypeObject_tp_as_number(Py_TYPE(obj));
-	return methods != NULL && PyNumberMethods_nb_index(methods) != NULL;
-
-}
 PyObject* _PySequence_ITEM(PyObject* obj, Py_ssize_t index) {
 	PySequenceMethods* methods = PyTypeObject_tp_as_sequence(Py_TYPE(obj));
 	return PySequenceMethods_sq_item(methods)(obj, index);

graalpython/com.oracle.graal.python.cext/src/bytesobject.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2018, 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2023, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The Universal Permissive License (UPL), Version 1.0
@@ -79,6 +79,9 @@ char* PyBytes_AsString(PyObject *obj) {
 
 int PyBytes_AsStringAndSize(PyObject *obj, char **s, Py_ssize_t *len) {
     *s = (char*)GraalPyTruffle_Bytes_AsString(obj);
+    if (*s == NULL) {
+        return -1;
+    }
     if (len != NULL) {
         *len = GraalPyBytes_Size(obj);
         return 0;
@@ -514,3 +517,17 @@ _PyBytesWriter_WriteBytes(_PyBytesWriter *writer, void *ptr,
 
     return str;
 }
+
+PyObject* bytes_subtype_new(PyTypeObject *type, int8_t* contents, Py_ssize_t n) {
+    PyObject* bytes = type->tp_alloc(type, n);
+    if (bytes != NULL) {
+        char* dst = ((PyBytesObject*)bytes)->ob_sval;
+        memcpy(dst, contents, n);
+        dst[n] = '\0';
+    }
+    return bytes;
+}
+
+void* PyTruffle_NativeBytesItems(PyBytesObject* bytes) {
+    return polyglot_from_i8_array((int8_t*)bytes->ob_sval, bytes->ob_base.ob_size);
+}

graalpython/com.oracle.graal.python.cext/src/capi.c

@@ -432,6 +432,12 @@ PyAPI_FUNC(RESULT) get_##NAME(RECEIVER obj) {                      \
 PyAPI_FUNC(RESULT) get_##NAME(RECEIVER obj) {                    \
     return obj->FIELD? obj->FIELD->NAME : NULL;                   \
 }
+
+#define PRIMITIVE_EMBEDDED_FIELD_GETTER(RECEIVER, FIELD, RESULT, NAME) \
+PyAPI_FUNC(RESULT) get_##FIELD##_##NAME(RECEIVER obj) { \
+    return obj->FIELD.NAME; \
+}
+
 TYPE_FIELD_GETTER(PyObject*, ob_type)
 PRIMITIVE_FIELD_GETTER(PyObject*, Py_ssize_t, ob_refcnt)
 PRIMITIVE_FIELD_GETTER(PyVarObject*, Py_ssize_t, ob_size)
@@ -514,6 +520,8 @@ PRIMITIVE_FIELD_GETTER(PyTypeObject*, unsigned long, tp_flags)
 PRIMITIVE_FIELD_GETTER(PyModuleDef_Base*, Py_ssize_t, m_index)
 PRIMITIVE_FIELD_GETTER(PyModuleDef*, Py_ssize_t, m_size)
 PRIMITIVE_FIELD_GETTER(PyModuleDef*, const char*, m_doc)
+PRIMITIVE_EMBEDDED_FIELD_GETTER(PyComplexObject*, cval, double, real)
+PRIMITIVE_EMBEDDED_FIELD_GETTER(PyComplexObject*, cval, double, imag)
 
 char* get_ob_sval(PyObject* op) {
 	return ((PyBytesObject *)(op))->ob_sval;
@@ -1633,6 +1641,10 @@ PyAPI_FUNC(int) PyIndex_Check(PyObject* a) {
 PyAPI_FUNC(PyObject*) PyInstanceMethod_New(PyObject* a) {
     return GraalPyInstanceMethod_New(a);
 }
+#undef PyIter_Check
+PyAPI_FUNC(int) PyIter_Check(PyObject* a) {
+    return GraalPyIter_Check(a);
+}
 #undef PyIter_Next
 PyAPI_FUNC(PyObject*) PyIter_Next(PyObject* a) {
     return GraalPyIter_Next(a);
@@ -2041,6 +2053,10 @@ PyAPI_FUNC(int) PyThread_acquire_lock(PyThread_type_lock a, int b) {
 PyAPI_FUNC(PyThread_type_lock) PyThread_allocate_lock() {
     return GraalPyThread_allocate_lock();
 }
+#undef PyThread_get_thread_ident
+PyAPI_FUNC(unsigned long) PyThread_get_thread_ident() {
+    return GraalPyThread_get_thread_ident();
+}
 #undef PyThread_release_lock
 PyAPI_FUNC(void) PyThread_release_lock(PyThread_type_lock a) {
     GraalPyThread_release_lock(a);
@@ -2277,6 +2293,10 @@ PyAPI_FUNC(PyObject*) _PyUnicode_AsUTF8String(PyObject* a, const char* b) {
 PyAPI_FUNC(int) _PyUnicode_EqualToASCIIString(PyObject* a, const char* b) {
     return Graal_PyUnicode_EqualToASCIIString(a, truffleString(b));
 }
+#undef _Py_GetErrorHandler
+PyAPI_FUNC(_Py_error_handler) _Py_GetErrorHandler(const char* a) {
+    return Graal_Py_GetErrorHandler(truffleString(a));
+}
 #undef _Py_HashDouble
 PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject* a, double b) {
     return Graal_Py_HashDouble(a, b);