Bug #1156

PyQt4 has been deprecated and is no longer available for download

Added by Joel Palmius almost 2 years ago. Updated almost 2 years ago.

Status:NewStart date:03/21/2017
Priority:HighDue date:
Assignee:Joel Palmius% Done:

0%

Category:MakeHuman GUI
Target version:MakeHuman 1.1.2

Description

Cross-posting from dev forum:

Riverbank computing, who have so far provided the binaries for PyQt4 for windows, have stopped providing these now. Their homepage states that PyQt4 is no longer supported and that everyone should upgrade to PyQt5.

While upgrading to qt5 is more than what is appropriate for a patch release, we still have to figure out what to do about it. For now my immediate solution is to use an old pyqt4 release I downloaded for the previous build machine.

For further reference:

pyside_pyqt.png (95.9 KB) Joel Palmius, 03/22/2017 11:14 AM

pyside_pyside.png (97.2 KB) Joel Palmius, 03/22/2017 11:14 AM

mh_hg_diagram.svg (508 KB) Jonas Hauquier, 03/23/2017 01:32 PM

screenshot_7_1490272231.png (314 KB) Jonas Hauquier, 03/23/2017 01:33 PM

screenshot_1_1490272647.png (304 KB) Jonas Hauquier, 03/23/2017 01:40 PM

screenshot_2_1490272713.png (278 KB) Jonas Hauquier, 03/23/2017 01:41 PM

screenshot_3_1490272795.png (345 KB) Jonas Hauquier, 03/23/2017 01:42 PM

mh_hg_diagram.svg (508 KB) Jonas Hauquier, 03/23/2017 01:59 PM

mh_hg_workflow.svg (105 KB) Jonas Hauquier, 03/23/2017 02:04 PM

screenshot_1_1490274391.png (314 KB) Jonas Hauquier, 03/23/2017 02:09 PM

1362
1363
1365
1366
1367
1368
1371

History

#1 Updated by Jonas Hauquier almost 2 years ago

We could also switch to pyside for the time being. It shouldn't be a lot of work porting.

#2 Updated by Jonas Hauquier almost 2 years ago

But qt5 porting, together with py3 compatibility should be the real way forward.
Still, the question whether we use riverbank's qt wrappers, or the fully open sourced pyside is open.

Edit: Pyside for qt5 is pyside2, which is actively maintained by the Qt project
Edit2: riverbank pyQt is not really a problem for us. Its free license is GPLv3, since MH is also (A)GPLv3 they are compatible. It's only if you want your project to be less restrictively licenced than GPL that you cannot use pyQt.

#3 Updated by Joel Palmius almost 2 years ago

Since I am essentially for having a less restrictive license in the future, I'm thinking this in itself is a very good reason to go pyside2.

The original pyside seems to have been essentially dropped. For example, there is no support for python versions after 3.4 (ie a python from 2014), and they talk on their bug tracker about having to abandon qt4 since it's no longer supported by qtcorp:

...this was last left as basically a wontfix, as one needs to modify both Qt 4.8 and PySide sources to compile with the compiler used for Python 3.5 on Windows. Furthermore, Qt 4.8 is no longer supported, and PySide should not be used for new projects. PySide2, the next version, is the future of Python and Qt, and supports Python 3.5.

So I don't think it's a viable solution to try to avoid a qt5 conversion by relying on the old pyside for now.

But qt5 porting, together with py3 compatibility should be the real way forward.

So, yes: It'd make more sense to go qt5 in the python3 branch since we're already doing disruptive things there.

#4 Updated by Jonas Hauquier almost 2 years ago

Does pyside2 still support python 2.7?

#5 Updated by Jonas Hauquier almost 2 years ago

Joel Palmius wrote:

yes: It'd make more sense to go qt5 in the python3 branch since we're already doing disruptive things there.

Not sure what you mean with disrupting, but I was hoping for a python2/3 compatible version of MH, instead of one that just completely ignored backward compatibility support. The python language has made it pretty simple and clean to make your code backwards compatible, and I think we should embrace that.

Of course that is given that third party libraries also maintain py2 support.

#6 Updated by Joel Palmius almost 2 years ago

My question is why we need to bother with the backwards compatibility at all? Py2 is approaching end of life. Ok, admittedly there's still some time, but it's going to happen in the foreseeable future.

If we drop py2 backwards compatibility we will be able to use new language features and, more importantly, not have to worry about testing against old python versions.

What do we gain by spending the extra effort required to maintain py2 compatibility? (This is not a rhetorical question, I'm curious about why we would want to do it in the first place)

#7 Updated by Jonas Hauquier almost 2 years ago

Joel Palmius wrote:

What do we gain by spending the extra effort required to maintain py2 compatibility?

My impression is that we will maintain support for more systems.
Also it could introduce a more systematic approach into the porting process, requiring more thought to be put into it and resulting in better code quality.
For example we need to get rid of the auto-generated py3 code in most cases.

#8 Updated by Joel Palmius almost 2 years ago

Which current systems are not able to use python 3? As an example, the LTS ubuntu 12.04 has reached end of life, and it shipped with python 3.2.

If the main reason for maintaining py2 compatibility is that it forces us write code in a certain manner, then in my thinking it isn't worth it. Not when it also means not being allowed to use new syntax and library features.

#9 Updated by Joel Palmius almost 2 years ago

For reference, python version by linux distro:

https://docs.google.com/spreadsheets/d/1JSX8fBmPb84emTmV0Kmyf0_r6R0kZM0h9Wdm91tn7Kg/edit#gid=0

Seems that all modern releases except RHEL/CentOS support at least python 3.4.

#10 Updated by Jonas Hauquier almost 2 years ago

Joel Palmius wrote:

Not when it also means not being allowed to use new syntax and library features.

I've maintained quite a bit of code in both python 2 and 3, and I must say that I don't find many new py3 language features that are really worth locking into one version.

Also keep in mind that many of these "python3" features are only introduced in a specific version, eg python 3.3 or 3.6. If you're locking down the compatible version to a version of python released a few years ago, you're probably going to exclude some systems.

What you're getting in newer versions of the syntax is mainly some added convenience libraries (which admittedly sometimes can be useful), but much of the focus is in my opinion to different application domains than MakeHuman. eg. parallelized applications, server infrastructures, non blocking i/o

#11 Updated by Jonas Hauquier almost 2 years ago

With regards to using python 3's new features: I think we should carefully evaluate the blind list() casts that 2to3 introduced everywhere in the port.
One of python 3's main features is to use iterators wherever possible, and that's a good thing. Because these converters have no context, they just do the thing that works every time: convert them all back to python2-style lists.

A human should go over them and remove where an iterator is acceptable (which is in most cases). This might improve performance, improves code quality and readability, and just makes the code look less ugly.

#12 Updated by Joel Palmius almost 2 years ago

So basically we have:

Pro keeping py2 compatibility:

  • MH might run on five year old linux distros and RHEL/Centos (if anyone is using those as desktop systems)
  • It might force us to spend extra effort writing code in a different manner

Con keeping py2 compatibility:

  • We'd have to maintain and test the said backwards compatibility
  • We have to make sure that we don't use anything more than what was state of the art 2010, whether it is useful or not
  • We still have to maintain the encoding mess present with py2, and add to that the differences in string management between py2 and py3

Sorry, even given the benefits you list, I still don't see how maintaining py2 compatibility could be a priority now. We don't need more work.

#13 Updated by Joel Palmius almost 2 years ago

Amusingly enough, things work better with pyside than with pyqt. I made a quick test with pyside (for qt4) for python3 (it's available as the "pyside" branch in the repo).

With PyQt4 on Aranuvir's branch:

With PySide on the pyside branch (branched from Aranuvir) :

The culprit seemed to be a strange cast where a byte array in the form of a memory view was cast to a string.

#14 Updated by Jonas Hauquier almost 2 years ago

Joel Palmius wrote:

The culprit seemed to be a strange cast where a byte array in the form of a memory view was cast to a string.

Interesting. Care to elaborate on that? eg line number, commit number

#15 Updated by Jonas Hauquier almost 2 years ago

Does this fix the problem for people that had gl rendering issues before? Or does this only fix a problem for people that had no gl issues with the stable mh?

Edit: looking again I guess I was a bit too enthusiastic, what we see in the screenshot is not the rendering issue we were experiencing, it's just "no texture" (well, actually the texture I choose to load explicitly when a texture is not found) that is rendered as litsphere map.

#16 Updated by Joel Palmius almost 2 years ago

No, this has nothing to do with the need of --noshaders on certain cards/platforms, at least not that I know of.

The fix was for getting a byte array out of a qtimage:

https://github.com/makehumancommunity/makehuman_stable_python3/commit/76481b05e76f8c8a329fc07daa38e4852017b0f8

I'm aware that there's no size limitation on the produced array here (as seems to have been with the asstring() conversion), but so far I've not noticed any downside of this.

#17 Updated by Aranuvir # almost 2 years ago

Not sure where to put bug reports for the Pyside transition, so I' writing the report here:

Traceback (most recent call last):
  File "./core/mhmain.py", line 540, in loadPlugin
    module.load(self)
  File "plugins/7_data.py", line 127, in load
    taskview = category.addTask(DataTaskView(category))
  File "plugins/7_data.py", line 92, in __init__
    self.tree = self.addLeftWidget(DataTree(G.app))
  File "plugins/7_data.py", line 51, in __init__
    self.item = self.addTopLevel('Application')
  File "./lib/qtgui.py", line 1820, in addTopLevel
    item = TreeItem(text, None, isDir)
  File "./lib/qtgui.py", line 1788, in __init__
    self.setIcon(0, TreeView._dirIcon)
TypeError: 'PySide.QtGui.QTreeWidgetItem.setIcon' called with wrong argument types:
  PySide.QtGui.QTreeWidgetItem.setIcon(int, NoneType)
Supported signatures:
  PySide.QtGui.QTreeWidgetItem.setIcon(int, PySide.QtGui.QIcon)

Searching for the declaration of QtGui.QStyle.SP_DirIcon, it resolves to:

SP_DirIcon = None # (!) real value is ''

#18 Updated by Jonas Hauquier almost 2 years ago

Joel Palmius wrote:

The fix was for getting a byte array out of a qtimage:

https://github.com/makehumancommunity/makehuman_stable_python3/commit/76481b05e76f8c8a329fc07daa38e4852017b0f8

Ah yes. In python2 string == bytes (a bytestring, basically a bunch of ascii codes representing the string, encoded in some encoding such as ascii or utf-8, or one of the many fantastic encodings windows uses), they are literally synonyms of each other. Whereas unicode is a bytecode presentation of a text string.
In python3 a string is implicitly that decoded unicode representation. So when you say "string" in python2, your're probably wanting to say: bytes in python3

That's not entirely true, as in MH some string() casts might have been used to cast from QString to python native string. Which would have been a bad thing, because really they should have been casts to unicode. (I can't remember if this is actually the case in the code, though).
There's another exception to this rule. When you're getting file paths, these will be bytestrings encoded in whatever the filesystem uses for encoding its filenames. Trying to decode these to unicode might be hazardous (as it seems we cannot reliably tell the encoding used. It's also a mystery whether we can pass unicode objects back to filesystem functions and if they will be encoded back using the proper encoding). If you get a file path through Qt libraries, you will get this as a Qstring, which will have to be encoded to either a unicode or string object (in python2), and a string or bytes object in python3 respectively.

Anyway, instead of fixing errors as they crop up, it might be better to check the entire python3 codebase for occurences of string( ( and bytes(? ) and decide what was meant there, a text string? (in which case we probably want a unicode, which is just a string in py3) effectively a binary bytestring? (py3 uses the bytes type for that).
For the contents of an image, for example, that's definitely going to be bytes

#19 Updated by Jonas Hauquier almost 2 years ago

Aranuvir (Moderator) wrote:

Searching for the declaration of QtGui.QStyle.SP_DirIcon, it resolves to:

SP_DirIcon = None # (!) real value is ''

What does that comment even mean?
Clearly the setIcon function is not satisfied with a None object.

Perhaps a fix could be to put this in module scope:

TreeView._dirIcon = ''  # pyside declares this as None but is not accepted by setIcon()

Or alternatively:

self.setIcon(0, TreeView._dirIcon if TreeView._dirIcon is not None else '')

#20 Updated by Aranuvir # almost 2 years ago

Hm, the problem is self.style().standardIcon(QtGui.QStyle.SP_DirIcon) is returning a QIcon, but the class attribute QTreeView._dirIcon remains None.

line 1813 in qtgui,py:

TreeView._dirIcon = self.style().standardIcon(QtGui.QStyle.SP_DirIcon)

#21 Updated by Jonas Hauquier almost 2 years ago

That seems like part of it. Added with diagrams made by Manuel. But no mention of the use of the bugtracker, or how feature branches are being merged.

The original must have used these graphics at some point

#22 Updated by Joel Palmius almost 2 years ago

To my knowledge, there is no other documentation page than this that mentions redmine. Looking at the old site (which I still have around), that old page reads the same as this one.

Anyway, please leave the bugtracker alone for a few hours while I try to manually edit the DB to extract issues from the malfunctioning subprojects.

#23 Updated by Jonas Hauquier almost 2 years ago

That didnt go as planned...

This image shows the makehuman main development repo against different independent feature branches, maintained in a separate repo (work happening on main branch in those repos). These features branches can be worked on independently, without suddenly disrupting the stability of the main development branch, and without delaying planned release dates.
While the main development branch contains tags (which correspond to milestones in the issue tracker), a feature branch just has one milestone, which is acceptance for mainline inclusion.
Each such feature branch can have its own sub project in the issue tracker, both to track its progress (an estimate toward merge milestone), have tickets separate from the main development tracking, and provide a link to which repository the code is stored in.
This results in an overview of all development happening on the MakeHuman project, from the main page of the issue (or development) tracker.

When a feature is deemed stable enough, it can be planned for inclusion in the mainline. It is planned for inclusion before a milestone (a feature release), when that release is the next one to come out and not already in feature lockdown. If the main branch is already in feature lockdown, meaning that a release tag and merge to stable is imminent, the feature will have to wait to be merged until after the next release.

Then the feature can be merged into main branch. In order to do that, all changes to main branch must be merged into the feature branch first (can be done continuously by the developer(s)). Bugfixes in the main repository can also happen against the stable branch, if they are only fixes to issues in the current stable version, and not add new features. These bugfixes are in turn merged back to the main (unstable) branch of the main repo, after which they can be merged into the main branch of the separate feature repository. The merge of the accepted feature into the main repository will happen into the main branch (unstable), which will become the new stable once it is released. The merge means that the feature will appear in the next upcoming feature release.
A feature ticket in the bugtracker on the main project can track the progress, link to the subproject, and can be closed and tagged with the proper version milestone to indicate when a feature branch was merged.
From these closed tickets on the roadmap, a changelog for the new release can be generated

#24 Updated by Jonas Hauquier almost 2 years ago

Most of the old documentation might be in these diagrams:

(mh_hg_diagram.svg )
This diagram shows how the main and feature repositories relate to each other, how dvcs works with developers local computers, and it shows how feature repositories (I call also them feature branches, but with hg it's recommended to use feature "clones" instead of branches. Mh only has two branches: main (unstable) and stable) relate subprojects in the issue tracker.

(mh_hg_workflow.svg )
This diagram shows how DVCS (distributed version control system, like hg or git) is used in practice, with both repositories stored publicly on servers and checked out locally on the computer of a developer. And shows the general workflow of interacting between them. This was meant to instruct people new to the concept of DVCS.

#25 Updated by Jonas Hauquier almost 2 years ago

A practical illustration of how one would go about starting their own feature branch (clone) for MakeHuman using bitbucket

We can probably distill a wiki article from these :)

#26 Updated by Joel Palmius almost 2 years ago

I have not seen any of these images before. Nor do I remember being part of any discussion where this was decided. Nor do I find any mention of them on the old site.

Where was all this posted?

#27 Updated by Jonas Hauquier almost 2 years ago

Hard to remember, it's been a while.
This was thorougly discussed on IRC, as I remember.
It's also been implemented since then for all large features and on the issue tracker (which is why you had the subprojects with attached repository).

We needed a structured way of keeping track of all features and their progress, while not burdening the mainline release by them. Basically any experiments that might break the main code for a while, or that imposed significant changes, needed to be outside of the main development, so that at any time we can call it a wrap and make a new release, even if that one feature was not completely finished yet.
We wanted to speed up development and releases this way, and wanted to get rid of the many unfinished experiments that littered the code before we took this approach. So a second reason was improving code quality, by allowing proper auditing and discussion before large features were accepted.

Examples of this are the commandline branch, which in its current state still introduces new bugs to MH, so should only go in when it is really fully working.
Also the new skeleton and posing/animation system was developed in this fashion: it consisted of a large change in code, a big refactor and introduction of new concepts and structures, introduced many things that at first were buggy. The feature was included when it was working properly, and became part of 1.1 if I remember correctly. It was listed as one of the big new features of that release (you can probably still find the feature ticket for it under the 1.1 milestone).

Hence, using only the standard versions for milestones in the issue tracker of the main project, which correspond to release tags in the hg repo, and which correspond to the version that is displayed when mh is started.
The python 3 porting is an excellent example of such feature branch, where for a while the application will work less well and stable than the mainline. When this is tested enough and deemed stable, it can be planned for inclusion for a version of MH. Without the need to already mark a version of MH as "python 3 port". It's the other way around, a certain version release of mh will contain the python 3 port as new feature.

#28 Updated by Joel Palmius almost 2 years ago

Can you check if the command line 2 project now contains the issues you remember from the old command line project? http://bugtracker.makehumancommunity.org/projects/cmdline2/issues

Also available in: Atom