Python 3 support

[city-in-the-sky]

Pattern should start supporting Python 3. Looking at the amount of code, it is a non-trivial task and any help is much appreciated.

[pemistahl]

Hi Tom,

I'm a graduate in computational linguistics and would like to contribute to Pattern. Can you be more explicit about how Pattern should support Python 3? That is, do you want to maintain two different branches in parallel, one for Python 2 and one for 3? Or do you want to have a single code base that works both with 2 and 3? In the latter case, a library such as six would be useful.

Let me know what you think.

Cheers,
Peter

[city-in-the-sky]

Hi Peter,

My goal would be to have a single code base that works with 2 and 3, but I have little experience with Python 3 so I don't know how feasible it is. In any case, the task is becoming more urgent so I will start looking into it more. I took a look at six which seems very useful. It's MIT-licensed so it could be included in Pattern.

Any help is appreciated! Let me know what you think.

Best,
Tom

[hayd]

👍 on a single codebase.

I think the first stage is to add travis for testing (I looks like you're missing a requirements.txt file, so I'm unsure what deps it's missing (?) ). Travis will really help with conversion (and ensuring it continues to work on multiple platforms).

• Do you want to continue supporting python 2.5? (Travis no longer supports it, 2.5 usage is pretty low and not sure which deps still support it...)
• Check which deps support python 3 (things in requirements.txt), fix them up if need be.
• Once that's hooked up, you could run a modernizer/futurize (see http://python-future.org/automatic_conversion.html#automatic-conversion ) on your code (or use six) and see what happens...

Happy to help if you can pass a requirements.txt.

[waylonflinn]

Got through the first two steps outlined by @hayd in this fork (repo has a requirements.txt and .travis.yml).

Some of the tests need to be excluded.

from test.py

# pattern.db tests require a valid username and password for MySQL.
# pattern.web tests require a working internet connection 
# and API license keys (see pattern.web.api.py) for Google and Yahoo API's.

Travis is just running python -m unittest discover -s test right now.

[waylonflinn]

Ran futurize on the codebase. Here are some preliminary findings:

1. Some of the bundled dependencies appear to have already been futurized (they contain from __future__ import and from future import statements). Now that we have pip and virtualenv does it make sense to unbundle these?
2. Unicode is used extensively throughout the codebase. I used from __future__ import unicode_literals in several places (mostly for raw string literals), but this should probably be handled more carefully in the long term.
3. Does it make sense to replace web.json.encoder with the standard library module? There was a section starting with the comment ## HACK: hand-optimized bytecode; turn globals into locals that I wasn't sure how to deal with and had to comment out.

I'm a bit new to python, so any feedback is appreciated. This is a beautiful library and I'd love to see it get the unicode love from python 3.

[hayd]

My 2cents:

• from __future__ and from future are perfectly fine.
• there may be a reason for the byte hack (is it faster than std library?)
• mysql tests are fine, you have to set up the db with travis (one line in .travis)
• travis internet tests also fine, although they may fail sometimes... worry about that later (when/if it becomes a problem e.g. decorator to skip test if there is connection exception or something).

Not sure what to do about API keys, was wondering what other modules e.g. pandas did for those parts... IIRC there may be keys you can use for testing of clipped results...

Perhaps it makes sense to create a PR for this and comment there, then you can comment on specific bits of code :) ... first pass tests then make pretty

[city-in-the-sky]

There's an "official" fork of Pattern with the specific aim of making it compatible with Python 3:
https://github.com/pattern3

The wiki has some more information:
https://github.com/pattern3/pattern/wiki

The compatibility update is supported by a grant from the Python Software Foundation. This money is to be divided among contributors. You can read the grant proposal here:
http://www.clips.ua.ac.be/media/Pattern-3-grant-proposal.pdf

The fork is initiated by myself, Waylon Flinn and David Branner. Everyone (Peter & hayd?) is welcome to join as admin of the project. As admin, you'll be able to edit anything so feel free to take initiative! (we do encourage pull requests, so we can keep track of who did what)

[hayd]

Happy to help with this, however when I tried (and trying again just now) running the tests I get a load of exceptions (python 2.7). I suspect this is just initial set up on my machine...

What do I need installed / setup to run the test suite (locally)?

Assuming fresh python install (or env) the following is failing:

git clone ...
cd pattern
python setup.py install  # this *ought* to install dependencies, but I don't think it does
nosetests  # this should sniff out and run all the tests, and does.

See to the travis run in the above fork: https://travis-ci.org/pinleague/pattern/builds/32799385 (this is the kind of thing that's failing though that's a couple of months old).

[city-in-the-sky]

Hi Andy,

My knowledge of Travis is zero, but different people including yourself have suggested it as a first step so I will examine it more closely. Looking at the output of the link you provided, these look like typical Python 2 vs 3 errors, e.g., using print stuff instead of print(stuff) and except Exception, e instead of except Exception as e. These are easy to fix, I previously used regular expressions to update them in the source code, but not yet in the unit tests. I'll look at updating the unit tests and push it to pattern3.

Best,
Tom

[hayd]

@tom-de-smedt Lots of stuff to migrated to python 3 but this can really only done with confidence once tests pass (and at the moment I can't get them passing either locally or on travis on python 2.7!!!).

At the moment they (the python 2.7 tests) fail with errors from the bottom of this page: https://travis-ci.org/pinleague/pattern/jobs/32799386. Any ideas why?

[pemistahl]

Hi Tom,

as I wrote at the beginning of this year, I'm still interested in contributing to pattern. However, I have not started yet because I didn't really know where to start. But now there exists a concrete plan and I would like to be part of it. I haven't written Python code for more than a year now but it should be easy for me to get into it again (I wrote a lot of Python code during my studies and I like the language very much). Last but not least, I have been out of the computational linguistics area since I started my current job a year ago, but it would be great to deal with that stuff again.

Some things are not yet clear to me:

1. You wrote that the fork should be made compatible with Python 3.3 but 3.4 has been released already. Shouldn't it be compatible with 3.4 then?
2. In the fork's wiki you wrote that the fork should be made compatible with both Python 2.7 and 3.3. But what's the point of creating a fork explicitly named pattern3 if it should still support Python 2.7? In my opinion, we can provide for a much cleaner and optimized code base if we completely drop 2.7 support. Then, the usage of libraries such as six would become obsolete. Of course, the downside of this approach is to maintain two separate code bases.

I cannot tell you yet which module I would prefer to work on. First, I need to take a look at the code again. I'm not sure though whether it's a good idea to have a lot of admins for the fork. Working with pull requests is much better anyway due to the reasons you mentioned.

[hayd]

This was partially my misunderstanding (!), just running nosetests ran the abstract test methods, which fail (at least that's part of it). cleaning these classes is probably a good thing to do anyways (they are in an "interesting" style... e.g. IMO the suite functions should go), I've cleaned up a little...

I had to capture a few actual test failures and some HTTP403Forbidden and HTTP404NotFounds. There's also a couple of proper errors (in python 2), which for now I'm skipping those tests, but they really need looking at, I've labelled them FIXME in my branch (should I PR to pattern3 or here once passing?)...

As I said above, it worth making necessary that these tests pass reliably in python 2 before even attempting to migrate to python 3 (otherwise it's shooting in the dark). That said, I think the issues I've found (and labelled FIXME) are minor (or at least I'm hopeful that's the case if someone can look at them who understands the codebase!).

See hayd/pattern@c5d9c23...ce1fe81 (and on travis https://travis-ci.org/hayd/pattern/builds/39245044, unfortunately not quite passing python 2.6 and 2.7, I may have to skip/fix a couple more? Some tests seem flaky - especially those that compare e.g. to 0.771!).

Note1: This allows the test suite to be run by simply calling nosetests (or py.test).

Note2: I'm skipping the mysql tests atm, but that's no biggie to fix just an install in the yml (our objective is for no tests to be skipped on travis), the others are more important, but I'm afraid I need a patterns expert to look at the FIXMEs!!

---

Just to clarify the objectives here:

• skip tests which fail (on python 2) and label them FIXME - this is mostly done
• have travis running successfully (with some skips) on python 2
• remove the skips (by fixing the bugs (?) for the FIXMEs)
• once all tests are running on travis (and locally on tox), migration can begin safely

[hayd]

To answer @pemistahl I don't think going fully py3 (and dropping support for py27 is (Edit: NOT) a good option for a library... for the next decade!). I would like to see a shared code base and drop support for python <= 2.5 (nearly every library is dropping python 2.5 support).

I'd really like to see pattern3 (once ready) merge upstream into pattern.

[pemistahl]

@hayd OK, I get your point. I'm okay with that. It just reminds me again of how unhappy I am about the Python 3.* transition in general across the Python community.

Another question @tom-de-smedt : If working with pull requests is the preferred way for contribution, then why did you create the pattern3 fork? Anyone who wants to contribute would create their own fork anyway. Wouldn't it be sufficient to simply create a branch here in the main repo for this purpose?