[GR-17357] Add Jython emulation flag to make imports and catching Java exceptions work more like Jython

PullRequest: graalpython/614

Author: timfel

CHANGELOG.md

@@ -15,8 +15,8 @@ language runtime. The main focus is on user-observable behavior of the engine.
 * Fix `dict.__contains__` for dictionaries with only `str` keys for subclasses of `str`
 * Support NumPy 1.16.4 and Pandas 0.25.0
 * Support `timeit` module
-* Support importing Java classes using normal Python import syntax when the package is known
 * Improve performance across many Python benchmarks
+* Add a new `--python.EmulateJython` flag to support importing Java classes using normal Python import syntax when the package is known and to catch Java exceptions from Python code
 
 ## Version 19.2.0
 

ci.jsonnet

@@ -27,6 +27,7 @@ local builder = import 'ci_common/builder.libsonnet';
         builder.testGateTime(type="tagged-unittest", platform="darwin", timelimit=const.TIME_LIMIT["2h"]),
         builder.testGate(type="svm-unittest", platform="linux"),
         builder.testGate(type="svm-unittest", platform="darwin"),
+        builder.testGate(type="unittest-jython", platform="linux"),
 
         // junit
         builder.testGate(type="junit", platform="linux"),

doc/CONTRIBUTING.md

@@ -98,3 +98,67 @@ For debugging java implemented code execute:
 
 The command will also start a debug server, which can be used in an IDE. If the IDE was initialized properly 
 by using the command mentioned above, the existing `GraalDebug` run configuration can be used to debug.
+
+### Advanced commands to develop and debug
+
+Here are some advanced commands to debug test failures and fix issues.
+
+First, we have three sets of unittests in the base repository:
+1. Our own Python-bases unittests
+2. JUnit tests
+3. Python's standard library tests
+
+To run the first, you can use this command:
+
+    mx python-gate --tags python-unittest
+
+If some of the tests fail, you can re-run just a single test like this,
+substituting TEST-PATTERN (and possibly the file glob on the third line) with
+the test you want to run. Note that you can insert `-d` to debug on the Java
+level or use `--inspect` to debug in the Chrome debugger.
+
+    mx [-d] python3 [--inspect] \
+        graalpython/com.oracle.graal.python.test/src/graalpytest.py \
+        graalpython/com.oracle.graal.python.test/src/tests/test_*.py \
+        -k TEST-PATTERN
+
+To run the JUnit tests, you can use this command:
+
+    mx python-gate --tags python-junit
+
+To run a subset of the tests, you can use the following. Again, you can use `-d`
+to attach with a Java debugger.
+
+    mx [-d] unittest JAVA-TEST-CLASSNAME
+
+To run the Python standard library tests, you can use the following:
+
+    mx python-gate --tags python-tagged-unittest
+
+Note that we use "tag files", small `.txt` files that select which tests to run,
+so we only run tests that we know should pass. To run a subset of those tests,
+use the following command. However, the way we run those tests is by spawning a
+sub-process for every stdlib tests, to avoid interference while our
+implementation isn't quite ready, so you have to put the flags somewhere else to
+debug. You can see `-debug-java` and `--inspect` below, to debug in Java
+debugger or Chromium, respectively.
+
+    ENABLE_CPYTHON_TAGGED_UNITTESTS=true mx python3 \
+        graalpython/com.oracle.graal.python.test/src/graalpytest.py \
+        [-debug-java] [--inspect] \
+        graalpython/com.oracle.graal.python.test/src/tests/test_tagged_unittests.py \
+        -k NAME-OF-CPYTHON-UNITTEST
+
+There's also multiple other gates that may fail with changes. One of these is
+our *style* gate, which checks formatting rules and copyrights. To auto-fix most
+issues, run the following command. Anything that's reported as error after this
+command you have to fix manually.
+
+    mx python-style --fix
+
+Another important gate is the gate that checks if you broke the native image
+building. To test if building a native image still works, you can use the
+following command. This will create a native executable called `graalpython` and
+print its path as the last output, if successful.
+
+    mx python-svm

doc/JYTHON.md

@@ -0,0 +1,80 @@
+# About Jython Compatibility
+
+Full Jython compatibility is not a goal of this project. One major reason for
+this is that most Jython code that uses Java integration will be based on a
+stable Jython release, and these only come in Python 2.x versions. The GraalVM
+implementation of Python, in contrast, is only targeting Python 3.x.
+
+Nonetheless, there are certain features of Jython's Java integration that we can
+offer similarly. Some features are more expensive to offer, and thus are hidden
+behind a commandline flag, `--python.EmulateJython`.
+
+## Importing Java classes and packages
+
+In the default mode, Java classes can only be imported through the `java`
+package. Additionally, only Java classes can be imported, not packages. In
+Jython compatibility mode, importing works more directly, at the cost worse
+performance for import failures in the general case.
+
+#### Normal mode
+
+Suppose you want to import the class `sun.misc.Signal`, normal usage looks like
+this:
+
+    import java.sun.misc.Signal as Signal
+
+For the `java` namespace, you do not have to repeat the `java` name (but can):
+
+    import java.lang.System as System
+
+#### Jython mode
+
+In Jython mode, Java packages can be imported and traversed, and they can be
+imported directly without going through the `java` module.
+
+    import sun.misc.Signal as Signal
+
+    import org.antlr
+
+    type(org.antlr.v4.runtime) # => module
+    org.antlr.v4.runtime.Token
+
+The downside of this is that everytime an import fails (and there are many
+speculative imports in the standard library), we ask the Java classloader to
+list currently available packages and traverse them to check if we should create
+a Java package. This slows down startup significantly.
+
+## Interacting with Java objects
+
+Once you get hold of a Java object or class, interaction in both modes works
+naturally. Public fields and methods can be read and invoked as expected.
+
+## Subclassing Java classes and implementing interfaces with Python classes
+
+This is not supported at all right now, there's no emulation available even in
+Jython compatibility mode. We have not seen many uses of this in the wild. Let
+us know if this is of interest to you!
+
+## Catching Java exceptions
+
+By default this is not allowed, because of the additional cost of checking for
+Java exceptions in the except statement execution. However, in Jython
+compatibility mode, the common case of catching a Java exception directly works:
+
+    import java.lang.NumberFormatException as NumberFormatException
+    import java.lang.Integer as Integer
+
+    try:
+        Integer.parseInt("99", 8)
+    except NumberFormatException as e:
+        pass
+
+Note that even in this mode, Java exceptions are never caught by generic except
+handlers, so this *will not* work:
+
+    import java.lang.Integer as Integer
+
+    try:
+        Integer.parseInt("99", 8)
+    except:
+        pass

graalpython/com.oracle.graal.python.cext/src/abstract.c

@@ -546,5 +546,5 @@ PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count) {
 
 UPCALL_ID(PySequence_InPlaceConcat);
 PyObject* PySequence_InPlaceConcat(PyObject *s, PyObject *o) {
-	return UPCALL_CEXT_O(_jls_PySequence_Concat, native_to_java(s), native_to_java(o));
+	return UPCALL_CEXT_O(_jls_PySequence_InPlaceConcat, native_to_java(s), native_to_java(o));
 }

graalpython/com.oracle.graal.python.test/src/graalpytest.py

@@ -60,9 +60,11 @@ class ThreadPool():
     if os.environ.get(b"ENABLE_THREADED_GRAALPYTEST") == b"true":
         maxcnt = min(os.cpu_count(), 16)
         sleep = time.sleep
+        start_new_thread = _thread.start_new_thread
         print("Running with %d threads" % maxcnt)
     else:
         sleep = lambda x: x
+        start_new_thread = lambda f, args: f(*args)
         maxcnt = 1
 
     @classmethod
@@ -73,7 +75,7 @@ def runner():
                 function()
             finally:
                 self.release_token()
-        _thread.start_new_thread(runner, ())
+        self.start_new_thread(runner, ())
         self.sleep(0.5)
 
     @classmethod

graalpython/com.oracle.graal.python.test/src/tests/cpyext/test_abstract.py

@@ -366,8 +366,8 @@ def compile_module(self, name):
 
     test_PyNumber_Lshift = CPyExtFunction(
         lambda args: args[0] << args[1],
-        lambda: ( 
-            (0, 0), 
+        lambda: (
+            (0, 0),
             (0, -1),
             (3, 2),
             (10, 5),
@@ -875,7 +875,7 @@ def compile_module(self, name):
     )
 
     test_PySequence_InPlaceConcat = CPyExtFunction(
-        lambda args: args[0] + args[1],
+        lambda args: args[0] + list(args[1]) if isinstance(args[0], list) else args[0] + args[1],
         lambda: (
             ((1,), tuple()),
             ((1,), list()),
@@ -895,4 +895,3 @@ def compile_module(self, name):
         arguments=["PyObject* s", "PyObject* o"],
         cmpfunc=unhandled_error_compare
     )
-

graalpython/com.oracle.graal.python.test/src/tests/test_interop.py

@@ -338,14 +338,24 @@ def test_java_imports():
             from java.util import ArrayList
             assert repr(ArrayList()) == "[]"
 
-            assert java.util.ArrayList == ArrayList
+            if sys.graal_python_jython_emulation_enabled:
+                assert java.util.ArrayList == ArrayList
 
-            import sun
-            assert type(sun.misc) is type(java)
+                import sun
+                assert type(sun.misc) is type(java)
 
-            import sun.misc.Signal
-            assert sun.misc.Signal is not None
+                import sun.misc.Signal
+                assert sun.misc.Signal is not None
 
+    def test_java_exceptions():
+        if sys.graal_python_jython_emulation_enabled:
+            from java.lang import Integer, NumberFormatException
+            try:
+                Integer.parseInt("99", 8)
+            except NumberFormatException as e:
+                assert True
+            else:
+                assert False
 
     def test_foreign_object_does_not_leak_Javas_toString():
         try:

graalpython/com.oracle.graal.python.test/src/tests/test_re.py

@@ -90,7 +90,8 @@ def test_none_value():
                     if special or text ])
     n = next(stream)
     assert not n[0]
-    assert str(n[0]) == 'None'
+    # GR-17928
+    # assert str(n[0]) == 'None'
 
 class S(str):
     def __getitem__(self, index):
@@ -457,5 +458,3 @@ def test_finditer_empty_string(self):
             r"(//?| ==?)|([[]]+)")
         for m in regex.finditer(''):
             self.fail()
-
-

graalpython/com.oracle.graal.python.test/src/tests/test_tagged_unittests.py

@@ -80,7 +80,12 @@ class TestAllWorkingTests():
 for idx, working_test in enumerate(WORKING_TESTS):
     def make_test_func(working_test):
         def fun(self):
-            cmd = [sys.executable, "-S", "-m", "unittest"]
+            cmd = [sys.executable]
+            if "--inspect" in sys.argv:
+                cmd.append("--inspect")
+            if "-debug-java" in sys.argv:
+                cmd.append("-debug-java")
+            cmd += ["-S", "-m", "unittest"]
             for testpattern in working_test[1]:
                 cmd.extend(["-k", testpattern])
             testmod = working_test[0].rpartition(".")[2]