[GR-61464] Minor docs updates.

PullRequest: graalpython/3648

Author: timfel

docs/contributor/IMPLEMENTATION_DETAILS.md

@@ -307,11 +307,14 @@ the next run of the cycle GC on the native side.
 
 On Linux, Python native extensions expect to lookup Python C API functions in the global namespace and specify no explicit dependency on any libpython.
 To isolate them, we copy them with a new name, change their `SONAME`, add a `DT_NEEDED` dependency on a copy of our libpython shared object, and finally load them with `RTLD_LOCAL`.
+The ELF format is not really meant to allow such modifications after the fact.
+The most widely used tool to do so, [patchelf](https://github.com/NixOS/patchelf), still sees regular bugfixes and can corrupt ELF files.
 
 On Windows there is no global namespace so native extensions already have a dependency on our libpython DLL.
 We copy them and just change the dependency to point to the context-local copy of libpython rather than the global one.
 
 On macOS, while two-level namespaces exist, Python extensions historically use `-undefined dynamic_lookup` where they (just like in Linux) expect to find C API functions in any loaded image.
 We have to apply a similar workaround as on Linux, copy to a new name, change the `LC_ID_DYLIB` to that name, and add a `LC_LOAD_DYLIB` section to make the linker load the symbols from our libpython.
+This is currently not fully implemented.
 
 Note that any code signatures are invalidated by this process.

docs/user/Native-Extensions.md

@@ -49,6 +49,11 @@ You can also manually trigger the detector with the Python `gc.collect()` call.
 
 ### Multi-Context and Native Libraries
 
+Using C extensions in multiple contexts is only possible on Linux for now, and many C extensions still have issues in this mode.
+You should test your applications thoroughly if you want to use this feature.
+There are many possiblities for native code to sidestep the library isolation through other process-wide global state, corrupting the state and leading to incorrect results or crashing.
+The implementation also relies on a venv to work, even if you are not using external packages.
+
 To support creating multiple GraalPy contexts that access native modules within the same JVM or Native Image, we need to isolate them from each other.
 The current strategy for this is to copy the libraries and modify them such that the dynamic library loader of the operating system will isolate them for us.
 To do this, all GraalPy contexts in the same process (not just those in the same engine!) must set the `python.IsolateNativeModules` option to `true`.

docs/user/Python-on-JVM.md

@@ -109,6 +109,7 @@ This enables you, for example, to use a Pandas frame as `double[][]` or NumPy ar
 ### Special Jython Module: `jarray`
 
 GraalPy implements the `jarray` module (to create primitive Java arrays) for compatibility.
+This module is always available, since we have not found its presence to have a negative impact.
 For example:
 
 ```python
@@ -142,10 +143,7 @@ However, implicitly, this may produce a copy of the array data, which can be dec
 
 ### Exceptions from Java
 
-To catch Java exceptions, use the `--python.EmulateJython` option.
-
-> Note: Catching a Java exception incurs a performance penalty.
-
+You can catch Java exceptions as you would expect.
 For example:
 
 ```python
@@ -161,8 +159,8 @@ For example:
 
 ### Java Collections
 
-* Java arrays and collections that implement the `java.util.Collection` interface can be accessed using the `[]` syntax. 
-An empty collection is considered `false` in boolean conversions. 
+* Java arrays and collections that implement the `java.util.Collection` interface can be accessed using the `[]` syntax.
+An empty collection is considered `false` in boolean conversions.
 The length of a collection is exposed by the `len` built-in function.
 For example:
 
@@ -186,7 +184,7 @@ For example:
     False
     ```
 
-* Java iterables that implement the `java.lang.Iterable` interface can be iterated over using a `for` loop or the `iter` built-in function and are accepted by all built-ins that expect an iterable. 
+* Java iterables that implement the `java.lang.Iterable` interface can be iterated over using a `for` loop or the `iter` built-in function and are accepted by all built-ins that expect an iterable.
 For example:
 
     ```python
@@ -242,12 +240,24 @@ For example:
 
 ### Inheritance from Java
 
-Inheriting from a Java class (or implementing a Java interface) is supported with some syntactical differences from Jython. 
-To create a class that inherits from a Java class (or implements a Java interface), use the conventional Python `class` statement: declared methods
-override (implement) superclass (interface) methods when their names match. 
-To call the a superclass method, use the special attribute `self.__super__`. 
+Inheriting from a Java class (or implementing a Java interface) is supported with some syntactical and significant behavioral differences from Jython.
+To create a class that inherits from a Java class (or implements a Java interface), use the conventional Python `class` statement.
+Declared methods override (implement) superclass (interface) methods when their names match.
+
+It is important to understand that there is actually delegation happening here - when inheriting from Java, two classes are created, one in Java and one in Python.
+These reference each other and any methods that are declared in Python that override or implement a Java method on the superclass are declared on the Java side as delegating to Python.
 The created object does not behave like a Python object but instead in the same way as a foreign Java object.
-Its Python-level members can be accessed using its `this` attribute. For example:
+The reason for this is that when you create an instance of your new class, you get a reference to the *Java* object.
+
+To call Python methods that do *not* override or implement methods that already existed on the superclass, you need to use the special `this` attribute.
+Once you are in a Python method, your `self` refers to the Python object, and to get back from a Python method to Java, use the special attribute `__super__`.
+And since we do not expose static members on the instance side, if you need to call a static method from an instance on the Java side, use `getClass().static` to get to the meta-object holding the static members.
+
+One important consequence of the two-object-schema here is that the `__init__` method on the Python object is actually called *before* the connection to the Java side is established.
+So you cannot currently override construction of the Java object or run code during initialization that would affect the Java half of the combined structure.
+You will have to create a factory method if you want to achieve this.
+
+For example:
 
 ```python
 import atexit
@@ -277,6 +287,8 @@ for record in handler.this.logged:
     print(f'Python captured message "{record.getMessage()}" at level {record.getLevel().getName()}')
 ```
 
+For more information about how the generated Java subclass behaves, see the [Truffle documentation](https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/TruffleLanguage.Env.html#createHostAdapter(java.lang.Object%5B%5D)).
+
 ## Embedding Python into Java
 
 The other way to use Jython was to embed it into a Java application. There were two options for such an embedding.

graalpython/com.oracle.graal.python.shell/src/com/oracle/graal/python/shell/GraalPythonMain.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2017, 2024, Oracle and/or its affiliates.
+ * Copyright (c) 2017, 2025, Oracle and/or its affiliates.
  * Copyright (c) 2013, Regents of the University of California
  *
  * All rights reserved.
@@ -47,6 +47,7 @@
 import java.util.Set;
 import java.util.UUID;
 import java.util.function.Function;
+import java.util.function.Predicate;
 
 import org.graalvm.launcher.AbstractLanguageLauncher;
 import org.graalvm.nativeimage.ImageInfo;
@@ -530,11 +531,17 @@ private String getLauncherExecName() {
      *            {@link #getEnv(String)} is used to retrieve $PATH.
      * @return The absolute path to the program or {@code null}.
      */
-    private String calculateProgramFullPath(String program, Function<Path, Boolean> isExecutable, String envPath) {
+    private String calculateProgramFullPath(String program, Predicate<Path> isExecutable, String envPath) {
         Path programPath = Paths.get(program);
 
         // If this is an absolute path, we are already fine.
         if (programPath.isAbsolute()) {
+            if (IS_WINDOWS) {
+                String resolvedProgramNameWithExtension = getProgramNameWithExtension(isExecutable, program);
+                if (resolvedProgramNameWithExtension != null) {
+                    return resolvedProgramNameWithExtension;
+                }
+            }
             return program;
         }
 
@@ -553,26 +560,15 @@ private String calculateProgramFullPath(String program, Function<Path, Boolean>
                     i = path.indexOf(File.pathSeparatorChar, previous);
                     int end = i == -1 ? path.length() : i;
                     Path resolvedProgramName = Paths.get(path.substring(previous, end)).resolve(programPath);
-                    if (isExecutable.apply(resolvedProgramName)) {
+                    if (isExecutable.test(resolvedProgramName)) {
                         return resolvedProgramName.toString();
                     }
 
                     // On windows, the program name may be without the extension
                     if (IS_WINDOWS) {
-                        String pathExtEnvvar = getEnv("PATHEXT");
-                        if (pathExtEnvvar != null) {
-                            // default extensions are defined
-                            String resolvedStr = resolvedProgramName.toString();
-                            if (resolvedStr.length() <= 3 || resolvedStr.charAt(resolvedStr.length() - 4) != '.') {
-                                // program has no file extension
-                                String[] pathExts = pathExtEnvvar.toLowerCase().split(";");
-                                for (String pathExt : pathExts) {
-                                    resolvedProgramName = Path.of(resolvedStr + pathExt);
-                                    if (isExecutable.apply(resolvedProgramName)) {
-                                        return resolvedProgramName.toString();
-                                    }
-                                }
-                            }
+                        String resolvedProgramNameWithExtension = getProgramNameWithExtension(isExecutable, resolvedProgramName.toString());
+                        if (resolvedProgramNameWithExtension != null) {
+                            return resolvedProgramNameWithExtension;
                         }
                     }
 
@@ -598,6 +594,28 @@ private String calculateProgramFullPath(String program, Function<Path, Boolean>
         return programPath.toAbsolutePath().toString();
     }
 
+    private static String getProgramNameWithExtension(Predicate<Path> isExecutable, String programStr) {
+        if (isExecutable.test(Path.of(programStr))) {
+            return programStr;
+        }
+        if (programStr.length() <= 3 || programStr.charAt(programStr.length() - 4) != '.') {
+            // program has no file extension
+            Path programNameWithExtension = null;
+            String pathExtEnvvar = getEnv("PATHEXT");
+            if (pathExtEnvvar != null) {
+                // default extensions are defined
+                String[] pathExts = pathExtEnvvar.toLowerCase().split(";");
+                for (String pathExt : pathExts) {
+                    programNameWithExtension = Path.of(programStr + pathExt);
+                    if (isExecutable.test(programNameWithExtension)) {
+                        return programNameWithExtension.toString();
+                    }
+                }
+            }
+        }
+        return null;
+    }
+
     private String[] getExecutableList() {
         String launcherExecName = getLauncherExecName();
         if (launcherExecName != null) {

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/shell/TestPathResolution.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2024, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2024, 2025, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The Universal Permissive License (UPL), Version 1.0
@@ -43,7 +43,7 @@
 import java.lang.reflect.Method;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.function.Function;
+import java.util.function.Predicate;
 
 import org.junit.Assert;
 import org.junit.Before;
@@ -57,11 +57,11 @@ public class TestPathResolution {
     @Before
     public void setup() throws NoSuchMethodException {
         // Use reflection to avoid exposing the method in public API
-        calculateProgramFullPathMethod = GraalPythonMain.class.getDeclaredMethod("calculateProgramFullPath", String.class, Function.class, String.class);
+        calculateProgramFullPathMethod = GraalPythonMain.class.getDeclaredMethod("calculateProgramFullPath", String.class, Predicate.class, String.class);
         calculateProgramFullPathMethod.setAccessible(true);
     }
 
-    public String calculateProgramFullPath(String executable, Function<Path, Boolean> isExecutable, String path) {
+    public String calculateProgramFullPath(String executable, Predicate<Path> isExecutable, String path) {
         try {
             return (String) calculateProgramFullPathMethod.invoke(new GraalPythonMain(), executable, isExecutable, path);
         } catch (Throwable e) {

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/objects/cext/copying/ElfFile.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2024, 2025, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The Universal Permissive License (UPL), Version 1.0
@@ -60,12 +60,20 @@ private String getPatchelf() {
         var executable = env.getPublicTruffleFile(context.getOption(PythonOptions.Executable).toJavaStringUncached());
         var patchelf = executable.resolveSibling("patchelf");
         var i = 0;
-        while (!patchelf.isExecutable() && i < path.length) {
+        while (!isExecutable(patchelf) && i < path.length) {
             patchelf = env.getPublicTruffleFile(path[i++]).resolve("patchelf");
         }
         return patchelf.toString();
     }
 
+    private static boolean isExecutable(TruffleFile patchelf) {
+        try {
+            return patchelf.isExecutable();
+        } catch (SecurityException e) {
+            return false;
+        }
+    }
+
     ElfFile(byte[] b, PythonContext context) throws IOException {
         this.context = context;
         this.tempfile = context.getEnv().createTempFile(null, null, ".so");