Bug #1184

Export not working with numpy 1.13

Added by TobiasWeis (Forum User) over 1 year ago. Updated 6 months ago.

Status:FixedStart date:07/06/2017
Priority:HighDue date:
Assignee:Joel Palmius% Done:

90%

Category:File export
Target version:MakeHuman 1.1.2

Description

THIS IS FIXED IN THE STABLE BRANCH ON BITBUCKET, BUT THE PPA HAS NOT YET BEEN UPDATED TO SUPPORT UBUNTU 18.04. IF YOU USE UBUNTU 18.04, RUN FROM SOURCE: http://www.makehumancommunity.org/wiki/Documentation:Running_MakeHuman_from_source, AND BE SURE TO CHECKOUT THE "STABLE" BRANCH

The export to any type is not working on my Ubuntu installation, resulting in the following error:

Exception during event onFileSelected
Traceback (most recent call last):
  File "./core/events3d.py", line 219, in callEvent
    method(event)
  File "./apps/gui/guiexport.py", line 112, in onFileSelected
    exporter.export(gui3d.app.selectedHuman, filename)
  File "plugins/9_export_collada/__init__.py", line 132, in export
    exportCollada(filename("dae"), cfg)
  File "plugins/9_export_collada/mh2collada.py", line 79, in exportCollada
    meshes = [obj.mesh.clone(config.scale, filterMaskedVerts=not config.hiddenGeom) for obj in objects]
  File "./core/module3d.py", line 160, in clone
    self.filterMaskedVerts(other, update=False)
  File "./core/module3d.py", line 311, in filterMaskedVerts
    inverse_uv_idx[uv_idx] = np.arange(self.texco.shape[0], dtype=np.int32)
ValueError: shape mismatch: value array of shape (21334,) could not be broadcast to indexing result of shape (14517,)

The first few lines of the system configuration are the following (please tell if you need the GL-versions etc.):

NO VERSION file detected, retrieving revision info from HG
Detected HG revision: r2133 (f1c1945ee9e3)
Initialized logging
VERSION: 1.2.0 (r2133 f1c1945ee9e3) [hg parents command]
HG REVISION: r2133 (f1c1945ee9e3) [hg parents command]
HG BRANCH: default
SHORT VERSION: v1.2.0
BASEMESH VERSION: hm08
IS BUILT (FROZEN): No
IS RELEASE VERSION: No
DEFAULT ENCODING: utf-8
FILESYSTEM ENCODING: UTF-8
STDOUT ENCODING: UTF-8
WORKING DIRECTORY: /home/weis/code/makehuman/makehuman
HOME LOCATION: /home/weis
PYTHON PATH: ./:./lib:./apps:./shared:./apps/gui:./core:/home/weis/code/makehuman/makehuman:/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/gtk-2.0
DLL PATH: /usr/local/cuda-8.0/bin:/home/weis/bin:/home/weis/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
SYS.VERSION: 2.7.12 (default, Nov 19 2016, 06:48:10)  [GCC 5.4.0 20160609]
SYS.PLATFORM: linux2
SYS.EXECUTABLE: /usr/bin/python
PLATFORM.MACHINE: x86_64
PLATFORM.PROCESSOR: x86_64
PLATFORM.UNAME.RELEASE: 4.8.0-58-generic
PLATFORM.LINUX_DISTRIBUTION: Ubuntu 16.04 xenial
NUMPY.VERSION: 1.13.0
No OpenGL_accelerate module loaded: No module named OpenGL_accelerate
Unable to load registered array format handler numeric
Unable to load registered array format handler vbo
Unable to load registered array format handler vbooffset
Gtk-Message: Failed to load module "overlay-scrollbar" 
PYQT.VERSION: 4.8.7
PYQT.JPG_SUPPORT: supported
PYQT.SVG_SUPPORT: supported
QT.PLUGIN_PATH: /usr/lib/kde4/plugins:/usr/lib/x86_64-linux-gnu/qt4/plugins:/usr/bin

On this debian installation, it works perfectly fine:

NO VERSION file detected, retrieving revision info from HG
Detected HG revision: r2133 (f1c1945ee9e3)
Initialized logging
VERSION: 1.2.0 (r2133 f1c1945ee9e3) [hg parents command]
HG REVISION: r2133 (f1c1945ee9e3) [hg parents command]
HG BRANCH: default
SHORT VERSION: v1.2.0
BASEMESH VERSION: hm08
IS BUILT (FROZEN): No
IS RELEASE VERSION: No
DEFAULT ENCODING: utf-8
FILESYSTEM ENCODING: UTF-8
STDOUT ENCODING: UTF-8
WORKING DIRECTORY: /home/weis/code_bernstein/makehuman/makehuman
HOME LOCATION: /home/weis
PYTHON PATH: ./:./lib:./apps:./shared:./apps/gui:./core:/home/weis/code_bernstein/makehuman/makehuman:/home/weis/pylearn2/src/pylearn2:/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/home/weis/.local/lib/python2.7/site-packages:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PILcompat:/usr/lib/python2.7/dist-packages/gst-0.10:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/lib/pymodules/python2.7:/usr/lib/python2.7/dist-packages/wx-3.0-gtk2
DLL PATH: /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SYS.VERSION: 2.7.9 (default, Jun 29 2016, 13:08:31)  [GCC 4.9.2]
SYS.PLATFORM: linux2
SYS.EXECUTABLE: /usr/bin/python
PLATFORM.MACHINE: x86_64
PLATFORM.PROCESSOR: 
PLATFORM.UNAME.RELEASE: 3.16.0-4-amd64
PLATFORM.LINUX_DISTRIBUTION: debian 8.7 
NUMPY.VERSION: 1.12.1
No OpenGL_accelerate module loaded: No module named OpenGL_accelerate
Unable to load registered array format handler numeric
Unable to load registered array format handler vbo
Unable to load registered array format handler vbooffset
PYQT.VERSION: 4.8.6
PYQT.JPG_SUPPORT: supported
PYQT.SVG_SUPPORT: supported
QT.PLUGIN_PATH: /usr/lib/x86_64-linux-gnu/qt4/plugins:/usr/lib/qt4/plugins:/usr/bin


Related issues

Related to MakeHuman - Bug #1213: No Exports Work Fixed 11/17/2017
Duplicated by MakeHuman - Bug #1227: Export Functionality Broken all exports Rejected 09/22/2018

Associated revisions

2137:91f776e01116
Added by Jonas Hauquier 9 months ago

Fix #1184 Ensure we stick to the length of the array that is assigned to

History

#1 Updated by Jonas Hauquier over 1 year ago

Might be the difference in numpy version.
Would you be willing to share a mhm save file that triggers this bug when exportrd to collada?

#2 Updated by Joel Palmius over 1 year ago

Which version of ubuntu?

And does the stable version work? (Seems you're running the "default" branch, ie unstable)

#3 Updated by TobiasWeis (Forum User) over 1 year ago

Hi, the Ubuntu version:
PLATFORM.LINUX_DISTRIBUTION: Ubuntu 16.04 xenial

Numpy might be ahead of the ubuntu release, I cannot recall whether I installed it from source or not.

Here is the mhm (it happens straight away, even with a totally fresh model):

# Written by MakeHuman 1.2.0 (r2133 f1c1945ee9e3) [hg parents command]          
version v1.2.0
tags mhm_test_fresh
camera 0.0 0.0 0.0 0.0 0.0 1.0
modifier macrodetails-universal/Muscle 0.500000
modifier macrodetails/African 0.333333
modifier macrodetails-proportions/BodyProportions 0.500000
modifier macrodetails/Gender 0.500000
modifier macrodetails-height/Height 0.500000
modifier breast/BreastSize 0.500000
modifier macrodetails/Age 0.500000
modifier breast/BreastFirmness 0.500000
modifier macrodetails/Asian 0.333333
modifier macrodetails/Caucasian 0.333333
modifier macrodetails-universal/Weight 0.500000
eyes HighPolyEyes 2c12f43b-1303-432c-b7ce-d78346baf2e6
clothesHideFaces True
skinMaterial skins/default.mhmat
material HighPolyEyes 2c12f43b-1303-432c-b7ce-d78346baf2e6 eyes/materials/brown.mhmat
subdivide False

#4 Updated by TobiasWeis (Forum User) over 1 year ago

Sorry, forgot. The stable version produces the same errors and does not export ( v1.1.1 (r2128 2d119491670c))

#5 Updated by Joel Palmius over 1 year ago

  • Status changed from New to Needs info

All this is very strange. I use 16.04 as my main platform, and MH was primarily built and tested on 16.04. I do not have any trouble loading and then exporting the MHM file above.

Numpy might be ahead of the ubuntu release, I cannot recall whether I installed it from source or not.

If you are in the habit of manually installing libraries built from source, I would lean towards blaming that. Especially if you installed a non-standard version of numpy.

Quite a lot of people use 16.04 together with MH, and this is the first we hear of this problem.

I realize this might be a big thing to ask, but could you temporarily move away the source version of numpy and see if it works if you install ubuntu's binary version?

#6 Updated by Aranuvir # over 1 year ago

  • Status changed from Needs info to Accepted
  • Target version set to MakeHuman 1.1.2

Well, the current numpy version on ubuntu 16.04 LTS is 1.11.0 (on my Lubuntu 16.04). After upgrading to numpy 13.1 I can confirm this issue. Solution for the OP: downgrade numpy to a version < 13.0.

I'm not going to close this task, since this will probably cause problems in the future...
BTW: Did anyone see any deprecation/future warnings before? Is this more a numpy bug?

#7 Updated by Jonas Hauquier over 1 year ago

Probably indexing behaviour changed a little in new numpy. We'll need a fix that preferably does not break older numpy versions for now.
I'll try to find a bit of time to look into.

#8 Updated by Jonas Hauquier over 1 year ago

  • Priority changed from Normal to High

#9 Updated by Rob Baer over 1 year ago

  • Related to Bug #1199: Error logs containing opengl failures added

#10 Updated by Joel Palmius about 1 year ago

  • Subject changed from Export not working in Ubuntu to Export not working with numpy 1.13

Another report from forums, for numpy 1.13.1:

Write MHX2 file /home/artista/makehuman/v1/exports/teste.mhx2
Exception during event onFileSelected
Traceback (most recent call last):
File "./core/events3d.py", line 211, in callEvent
method(event)
File "./apps/gui/guiexport.py", line 112, in onFileSelected
exporter.export(gui3d.app.selectedHuman, filename)
File "plugins/9_export_mhx2/__init__.py", line 55, in export
mh2mhx2.exportMhx2(filename("mhx2"), cfg)
File "plugins/9_export_mhx2/mh2mhx2.py", line 52, in exportMhx2
meshes = [obj.mesh.clone(cfg.scale, True) for obj in objects]
File "./core/module3d.py", line 160, in clone
self.filterMaskedVerts(other, update=False)
File "./core/module3d.py", line 311, in filterMaskedVerts
inverse_uv_idx[uv_idx] = np.arange(self.texco.shape[0], dtype=np.int32)
ValueError: shape mismatch: value array of shape (21334,) could not be broadcast to indexing result of shape (9416,)

http://www.makehumancommunity.org/forum/viewtopic.php?f=3&t=14913

#11 Updated by Joel Palmius about 1 year ago

Release notes for numpy 1.13 is here: https://docs.scipy.org/doc/numpy/release.html#numpy-1-13-0-release-notes

Release notes for numpy 1.12 is here: https://docs.scipy.org/doc/numpy/release.html#numpy-1-12-0-release-notes

I'm not versed enough in numpy to see from the notes if something we use have changed since 1.11.0 (which is known to be working).

#12 Updated by Joel Palmius about 1 year ago

The relevant code block is this:

    # Filter out and remap unused UVs
    fuvs = self.fuvs[self.face_mask]
    uv_idx = np.unique(fuvs.reshape(-1))
    inverse_uv_idx = - np.ones(self.texco.shape[0], dtype=np.int32)
    inverse_uv_idx[uv_idx] = np.arange(self.texco.shape[0], dtype=np.int32)
    for i in xrange(self.vertsPerPrimitive):
        fuvs[:,i] = inverse_uv_idx[fuvs[:,i]]

#13 Updated by Jonas Hauquier about 1 year ago

Most likely it's just a tiny difference in how the new version of numpy outputs the intermediary result.
Probably something a .reshape() can easily fix.
Will try to look at this in the near future, but at this time it's hard to say when I will have more time for MH.

#14 Updated by Rob Baer about 1 year ago

A pull request for an ostensible fix for this bug in numpy usage was posted to Bitbucket. Could somebody with appropriate Bitbucket clearance take a look at it. If it is good to go, we also want to clean up the Python 3 version.

#15 Updated by Joel Palmius about 1 year ago

  • Assignee changed from Jonas Hauquier to Joel Palmius

So far it works with numpy 1.12.1. I'll take a look at it for py3 and py2 + numpy 1.13 too.

#16 Updated by Joel Palmius about 1 year ago

  • Status changed from Accepted to Fix exists, needs testing

I've merged this (to default) now. I have also ported it to the master branch on github.

Remains to decide if it should be patched into bitbucket stable too.

I'm going to leave this issue semi-open for now, until decided what to do about stable.

#17 Updated by Jonas Hauquier about 1 year ago

I'll try to have a look at it this weekend, no promises though.
To make things easier, is it possible to link the related commits to this issue, like we did before on this tracker?
If it's not possible using redmine features, then just paste the hashes of the changesets in the comments.

#18 Updated by Jonas Hauquier about 1 year ago

Meh, sorry for not reading thoroughly enough, it was a pull request.
If someone can confirm that indeed it fixes the bug, then I'll cherry-pick it to stable branch.

For reference, this is the pull request:
https://bitbucket.org/MakeHuman/makehuman/pull-requests/33

Thanks Joel for merging it.

#19 Updated by Joel Palmius about 1 year ago

This is tested for not causing any trouble with numpy 1.12.1 or less on linux and windows, both py2 and py3.

It is tested to solve the problem and work with numpy 1.13.* and python3 on windows (without it FBX crashes with wrong shape, with it that crash doesn't appear).

It has not yet been tested with numpy 1.13.* and python2 on any platform.

NumPy 1.13 is still a pretty rare animal. Primary reason you'd get it is if you install numpy via pip. Artful (ubuntu 17.10) shipped with NumPy 1.12.1 less than a week ago.

#20 Updated by snisioi (Forum User) about 1 year ago

I'm using Ubuntu 17.04 x64 and tested this change with python2 and numpy>=1.13. For my machine learning projects I'm usually using the latest numpy checkout (1.14*) and it seems to work with that too.
I stumbled over this problem and thought to give it a fix since more recent numpy versions are more strict with assignments.
For example, in numpy 1.12, one can make these weird assignments:
>>> x = np.array([1,6,7,4,5,8])
>>> x0,2 = np.array([9,9,9,9])
>>> x
array([9, 6, 9, 4, 6, 8])
and in more recent versions these are forbidden, the indices must match the length of the assigned value.

#21 Updated by Jonas Hauquier about 1 year ago

  • Related to Bug #1213: No Exports Work added

#22 Updated by Jonas Hauquier 9 months ago

I have taken up the task of fixing this important bug

I'm attempting a new way of reporting about and investigating issues. I'll do this with a minimal illustration that can be run independently of the gui with the commandline branch of MH

#!/usr/bin/python2.7

import makehuman

makehuman.set_sys_path()
import headless

from core import G

def setup_app():
    G.app = headless.ConsoleApp()
    human = G.app.selectedHuman

setup_app()

h = G.app.selectedHuman
m = h.mesh

n = m.clone(filterMaskedVerts=True)

from IPython import embed
embed()

This is enough to reproduce the bug with numpy 1.13 installed, at the clone statement.

The nice thing is, now you have an application that boots instantly and can reproduce the bug without a need for clicks.
Even better, now that you don't have a main loop going, you can invoke IPython.embed() at any line in the code.
For example, I can put the lines

from IPython import embed
embed()

directly before the line where the bug is getting triggered in module3d.py
This gives me free access to interactively explore the stack and experiment, right before the moment where the bug happens.

#23 Updated by Jonas Hauquier 9 months ago

  • % Done changed from 0 to 90

Applied in changeset commit:makehuman_hg|91f776e01116.

#24 Updated by Jonas Hauquier 9 months ago

Proposing fix 91f776e01116feb45c0d009e7b6cf8a27d20d298
Can someone else verify this fixes it?

Are there any other issues caused by new version of numpy?

#25 Updated by Jonas Hauquier 9 months ago

For documentation completeness, I'll share how I debugged this issue.

I still have old numpy 1.9.2 on my system, which I didn't want to remove to be able to compare.
So I installed a recent numpy in a non-system folder:

pip install --target=/home/jonas/numpy-new  numpy

Then to make use of this version, I simply had to make sure it was in the python path, before the location of the system-installed version of numpy. To that end, I updated the corresponding line in makehuman.py (set_sys_path()):

syspath = ["/home/jonas/numpy-new", "./", "./lib", "./apps", "./shared", "./apps/gui","./core"]

#26 Updated by Jonas Hauquier 9 months ago

We really need unit testing for this kind of core functionality.

#27 Updated by Joel Palmius 9 months ago

Preliminary tests shows it works with both numpy 1.14.0 and 1.11.0 (ie before and after the problem arose).

I have tested with bombimg the exports with everything (clothes, topologies, rigs etc).

I have only tested obj and dae so far though.

#28 Updated by Jonas Hauquier 9 months ago

I tested with an OBJ export of a model with clothes that have face masking on the human mesh (thus UV coordinates that are not at the back of the buffer are filtered out, so there is a need for UV index remapping).
Importing in Blender, I can verify that indeed the UV indices of the human mesh are correct.

I think this one is solved.

#29 Updated by kshepherd (Forum User) 7 months ago

This bug is present in the latest Ubuntu 18.04 Bionic Beaver distro, updated to the latest versions 18 May 2018.
Whenever an attempt to export a model (at least dae format, others untested).

Exception during event onFileSelected
Traceback (most recent call last):
File "./core/events3d.py", line 211, in callEvent
method(event)
File "./apps/gui/guiexport.py", line 112, in onFileSelected
exporter.export(gui3d.app.selectedHuman, filename)
File "plugins/9_export_collada/__init__.py", line 132, in export
exportCollada(filename("dae"), cfg)
File "plugins/9_export_collada/mh2collada.py", line 79, in exportCollada
meshes = [obj.mesh.clone(config.scale, filterMaskedVerts=not config.hiddenGeom) for obj in objects]
File "./core/module3d.py", line 160, in clone
self.filterMaskedVerts(other, update=False)
File "./core/module3d.py", line 311, in filterMaskedVerts
inverse_uv_idx[uv_idx] = np.arange(self.texco.shape0, dtype=np.int32)
ValueError: shape mismatch: value array of shape (21334,) could not be broadcast to indexing result of shape (14517,)

Makehuman version 1.1.1-1
python-numpy version 1:1.13.3-2ubuntu1

I guess its back to using development versions.

I am happy to run tests and post logs if anyone wants them.

#30 Updated by Joel Palmius 7 months ago

  • Description updated (diff)

This is fixed in the stable version found at https://bitbucket.org/MakeHuman/makehuman. Be sure to check out the "stable" branch.

The PPA builds have not been updated for ubuntu 18.04, and won't work.

For instructions on running from source, see the linux section a bit down on http://www.makehumancommunity.org/wiki/Documentation:Running_MakeHuman_from_source

#31 Updated by Aranuvir # 6 months ago

  • Status changed from Fix exists, needs testing to Fixed

I didn't see any further crashes and consider the bug fixed.

#32 Updated by Aranuvir # 6 months ago

  • Related to deleted (Bug #1199: Error logs containing opengl failures)

#33 Updated by Aranuvir # 3 months ago

  • Duplicated by Bug #1227: Export Functionality Broken all exports added

Also available in: Atom